Lessons Learned 2021 Q4: challenges faced by a Machine Learning Engineer

Tech 💾, Business 📈 and Personal 👨‍💻. It's all in there.

Andrei Pascanean
Apr 7, 2022 11:49:31 AM

We all make mistakes, we all learn something new. This is my first lessons learned blog post, looking at what I learned in the last three months of 2021.

I’m a machine learning engineer with one year of experience working as a consultant at Vantage AI. Right now I’m working on an end-to-end ML project in the energy sector.

I’ll be making these blog posts regularly throughout the year, to reflect on my experience and make sure I don’t forget the lessons I learned. Hopefully I can use these lessons to become a better consultant, ML engineer and team player.

I hope these blogs can offer you tips and tricks for navigating your own career’s waters.

Keep learning and keep growing. One lesson at a time.

The Tech

#1: Deploy using Ansible

I started using Ansible for my project, which helps my team deploy our work on the client’s server programmatically. An especially useful Python package when there is no cloud environment available, which is the case on our current project. 

Comparable to a recipe, an ansible-playbook will provision your server environment the exact same way. Every. Single. Time. Think of it like a docker-compose but then for an entire operating system with apps.

The package itself is somewhat complicated to get into, but once you get the basics it’s easy to keep learning. You can keep it simple and only set up a basic environment, or take it all the way to encrypting files with ansible-vault.

#2: Stash docker images on Canister

We chose Canister as the place to host all of our docker container images. It allows you to open a registry that can house up to 10 docker images — free of charge! Most container registries are paid, although your client may already have a registry in place, such as a Gitlab/Github repository.

Again useful for when you don’t have a cloud environment that can offer the service. Push your images to the registry and docker pull them anywhere you need them. Simple. 

#3: Serve models with FastAPI

If you’re anything like me, and that’s not too high a bar to clear, you probably used to deploy APIs through a Flask server. Well I got an introduction to FastAPI and I was immediately sold. Awesome Python package for quickly building an API interface for your app or machine learning model. Minimal tinkering and comes with a Swagger-like frontend, free of charge.

This tool is especially useful if you need to demonstrate how to API works to your client. It comes with a fully interactive frontend that allows you to execute API calls directly in your browser. 

Besides the API response, it will also offer you the curl command coinciding with the call. I often use curlconverter if my client uses a different programming language than me.

The Business

#1: The client chooses the KPI

I had to be reminded that the client chooses the KPI on any project you work on. Outside of consultancy that ‘client’ can be any stakeholder that will evaluate your performance. In the end the client is the one who will determine your performance, so follow their metric! 

You can do this by aligning you model’s metric to a business KPI the client deems important. It’s usually the measure used to decide your project’s success and knowing it will help you a lot.

Our client’s KPI was the amount of revenue earned by trading on the energy market, using our model’s predictions. We instead were focused on metrics like the Mean Absolute Error, disregarding the client’s intended trading strategy. Lesson learned.

#2: Estimate task durations with a buffer

When presenting plans for the coming weeks to stakeholders, one question you are bound to get asked is ‘how long will it take?’. Whatever your way of counting required work-hours is, be sure to estimate with a buffer. 

Deploying your code to production going to take an estimated 3 days? Add on one more day just for safety — worst case you get to spend an extra day fixing bugs. Best case is you deliver early, always a nice bonus.

#3: Use the wisdom of the crowd

Always get a second opinion, and then a third. And possibly a fourth as well. Those with more or different experience give priceless advice, so use the wisdom of the crowd. 

This doesn’t only have to be about small details in your code or slides you’re making — others can offer useful tips on project roadmaps, deadlines and even approaches. 

Most of us have colleagues around us who have faced the same challenges we face today. The best part about working as a consultant, is that I’m always surrounded by people with a broad range of experience.

I learned that my problems are not unique, someone is bound to have had the same issues in the past. And of course, there is always more than one solution to a problem. So be sure to consult your office neighbour.

Always get a second opinion, and then a third. And possibly a fourth as well.

The Personal

#1: Always be writing

During meetings I kept telling myself ‘that’s important, make sure you don’t forget’. I also noticed that I kept forgetting more than I remembered. That’s why I now tell myself to always be writing, whether it’s meeting notes, cool ideas I hear or useful tips I hear. 

Take note of whatever is being discussed, you will probably forget it in an hour. The best part about taking notes is that you allow yourself to forget, it’s all on paper anyway. 

Be sure to regularly come back to and structure your notes to keep them readable for future you. I know this one is easier said than done, but I find it easiest to edit digitally. So pick out a notes app — I’m a fan of Apple’s — and tap away during calls. 

Pro tip from Sophron Vermeij: taking notes forces your brain to collect and then summarise everything you just heard, allowing you to remember everything better.

#2: Let others take the lead

I’m someone who enjoys taking the lead on a project and planning out everything that needs to happen. However, it’s important to also let others take the lead and make decisions. 

Allowing teammates to speak up and plot the way forward lets your team be exposed to more approaches for solving a problem. Different opinions and solutions are always good to consider, be sure to give others the space to do it.

I thought my approach was best, until we had to change course after one of our models failed to deliver. That’s when I realised that my colleagues had plenty of good ideas about how to solve the problem at hand, I just failed to listen to them earlier on.

A good way to keep to this tip is to always start a brainstorm session by asking others for ideas, instead of proposing them yourself.

#3: Hit the pause button

Work is work and leisure is leisure. The two don’t usually mix together nicely, at least not for long anyway. If you don’t keep the two separate then both can suffer. 

I was reminded that it’s important to hit the pause button and clock out with some time to myself after work. Recently our models were not doing so well in production. It was mostly due to our previous failure to use an accurate metric, but we just couldn’t seem to figure out the problem.

This pushed me into a frenzy to try and figure out how to fix our underperforming models. I was up late reading research papers, training models and regularly checking on the model’s performance — even on weekends!

Taking the time to recharge and refresh, worry free, is important for being productive the next day. I think this is a lesson that will always be relevant, as I try to balance my work and my down time.

In Review

A productive quarter with some cool experiences. Nice tools for deploying all that code, important rules when working with a client and some personal mistakes to learn from.

Hope you enjoyed reading! Smack that follow button if you did 👨‍🌾

P.S. I would love to hear your personal lessons learned, so feel free to leave a comment about your experiences!