Liopic on Tech – software engineering for evolved monkeys

Programming sound effects

Some months ago I bought a small Korg NTS-1, a digital synthesizer that can host 3rd party code both for oscillators and effects. Despite I’ve been always interested in the subject, from a programming perspective, I’ve never tried to program my own sound effects (except for a lab practice during my master years).

I’ve had the chance to play with its SDK in recent holidays, with some interesting results (a repo with some fx). It looks like there are a lot of techniques to learn, so I’m studying them while I mix what I’ve learn with some personal ideas.

Here I’m listing a growing collection of internet resources on the subject:

Martin Finke’s Making Audio Plugins, with a nice step-by-step module creation (for a VST plugin). I do like the filter section.
Musicdsp.org, a wide collection of code snippets with various interesting subjects.
Earlevel blog, with amazing articles on digital audio. The one on state variable filter was mind-opening for me.
The full category on audio effects at Wikipedia. Also the page on comb filter. And a nice page on phase modulation.
Stanford’s Introduction to Digital Filters. Lots of information on filters.
Digital Music Programming II lab, from Johns Hopkins University, with several interesting easy-to-understand code.
Digital Audio Effects from Stanford (pdf). Common guitar FX.
VCV Rack oscillator demo. Explains also how to fix common problems from digital oscillators.
VCV Rack DSP pages.
Pitchshifter on guitar.
Modeling an Analog Delay in Web Audio API.
Making of the VCV Drums.

Also some links that are specific to this synth SDK:

Making custom effects. Simple tutorials from Korg.
Videos exploring oscillators.
Centrevillage plugins’ code.
Creating custom oscillators and effects, from absolute music.

Anything missing that should be here?

Update: I’m adding the most complete forum on developing with Korg Logue SDK.

Mentoring juniors and interns remotely

My background

In my previous job I’d worked remotely during 5 years, and now due to COVID 19 we have to work from home again. From time to time people ask me about the lessons learned during my remote position: I mostly answer explaining the common problems and ways to solve them. Actually I even did a presentation on the subject.

However there is a topic that rarely appears in questions, and due to this pandemic situation, I consider vital to keep out IT industry in good shape: mentoring juniors and interns. In my experience, I mentored people from Mexico, India and Spain, remotelly.

Juniors in the remote world

The main problem working from home is the lack of communication. This problem has effect in all employees, but it’s far more severe in the case of juniors.

Juniors are basically novices in the Dreyfus model of skills, which means that they should be learning following rules and examples. Novices don’t know why an error arises. They look for constant feedback. Feedback is key to improvement, which is what you want a junior to focus on. Feedback is communication.

Juniors can’t work autonomously. They need the guide of a senior. They may not need the big picture, but a clear task to do, challenging but understandable: they need to keep momentum. They need a plan with examples. In some cases, like working in a fast-moving startup, the lack of plan can be very confusing.

Juniors bring freshness and enthusiasm. But if they don’t feel on track, their excitement starts to go down; and believe me, it’s the worst that can happen. It’s important to realize this situation as soon as possible, because depending on junior’s personality (openness, sincerity, etc), this problem could remain hidden for weeks or even months, and leads to fatal consequences.

There is a minor problem too that use to happen in remote situations: the lack of “team” feeling. In the case of juniors this can be an issue too.

Communication

Communication is the key to keep a junior on track: it helps his learning process and his daily rhythm. But working from home makes this communication less casual, so we need to patch this problem with some processes. Following I present some possible solutions.

Daily Learning

A senior should be always available to give feedback to a junior. Moreover, it should make him understand that if he has any doubts, he can freely ask about it, despite he could sound silly. This also helps to give advice just-in-time, while keeping the feedback loop really short.

It’s is vital that each conversation with a junior ends with an explicit “I understand” or some other form of ACK. A quiet answer is no answer.

A senior should give examples to learn from, and take always the chance to explain some lessons learned while doing a similar task (that the junior is currenlty doing).

Finally, a senior should challenge the mind of a junior. A common way is to ask him for alternative solutions: try to find 2 solutions to the problem (or at least a variation of the solution); and later discuss the trade offs together.

Planning

It’s mandatory to set 1:1’s frequently. In these meeting there should be a follow up of the junior’s career evolution. Your company may have a career track, like this one by patreon, or just a simple list of concepts to cross out. But anyway the junior should understand clearly his good points and the ones to improve.

Junior members should know the general idea of the following tasks, and have at clear path: a list of tasks with easy explanations.

Team building

There are lots of different actions for team building, even for the remote case. But these days it’s quite common to see online meetups and tech events, some of them with a cheap or free access. It’s a great chance to invite juniors to watch the event together, and later discuss the subjects.

Another similar approach is a reading club: find some must-read tech book, and set a time to discuss each chapter together.

Final words

This new work-at-home world has come to stay. Hiring and mentoring juniors can be more difficult, but the energy and momentum that bring to the team is worth of. And, frankly, they are cheaper to hire too. The industry can’t live without them, and now it’s not the time to look away.

Be the mentor you wanted to meet when you were a novice.

This text was proofread by some juniors.

4 things I would’ve liked to know about Cloud Functions

We had an important project to do: build and keep updated some transformed tables in BigQuery with data that comes from a transactional system. We needed 3 pieces: the code that build the tables, a place to deploy this code and a scheduler to call it. Given the code piece was in Python, we had to evaluate different platforms to deploy.

The platform chosen was Google Cloud Functions. Actually there is a nice diagram to let you choose among Google services that helped out (but for some reason it was not easy to found). We could’ve tried it deploying on our own hardware server, but a lateral idea with this project was testing the cloud.

Cloud Functions are perhaps the easiest way to deploy your code: given it’s Python, Go or Node.js; for other languages you could try the new Cloud Run, which is basically a docker container that is executed on an external event. A Cloud Function can be triggered using a HTTP call, a Google PubSub message, or some platform internal events (like when a file is uploaded). In our case, we use Google Scheduler (a basic cron) that sends a PubSub message.

Up to here everything sounds perfect, but we found 4 minor problems due to the lack of knowledge of the platform:

DEPLOY AND LOGS WITH DELAY
As you deploy and use the Function on the cloud, the results will come with delay. This is not a real problem, but if you’re get used to work with “tail” or other console scripts to explore the logs, you have to relax and wait a couple of minutes before sentencing your code is not working.

TIMEOUT
Cloud Function’s documentation says that this product is intended to short-running scripts, so it comes with a 1 minute timeout. We missed this detail at first, and the actual use misled us to think there was no timeout: if you manually launch the Function, apparently there is no timeout and runs for several minutes. But later, having a look at the logs, we found the timeout error (that actually is logged as INFO instead of ERROR!). It seems that when our script uses more than 1 minute, it continues until somebody else asks for resources: so if we are in a relaxed pool, we can be lucky and run it more time.

However you can configure, on deploy time, the timeout limit up to 9 minutes. Unluckily our code runs for 11 minutes, so we had to split it.

NO WRITE PERMISSIONS
The library we use does some disk-writing internally, and we didn’t realize it until we saw the problem in the logs. The good thing is that you can just write in /tmp, so we only had to reconfigure the library to write there. The weird thing is that anything that your code writes into /tmp is also written in the function log, so the logs can become difficult to follow.

SINGLE THREAD
This was the most trickiest one! We are using a Python library that, by default, creates 4 threads of execution. For some reason this doesn’t work well on Cloud Functions, and sometimes the connection with BigQuery is closed before all threads have finished. So we had to use an undocumented feature of the library to work only with a single thread.

Summing up, Google Cloud Functions is a lightweight way to execute your Python scripts, with a really way to deploy and use. But sometimes things go wrong under the hood, so you should READ THE LOGS to find out if all is ok. Checking that the final results matches what you expect will help too (for instance, doing automated test-queries that try to find non matching numbers).

Disclaimer: we chose Google Cloud Functions due to the particular background of the company (team, knowledge, etc). Depending on the task to do, you might want to have a look at other more specific products, that could help you better to make ETLs or processing data (for instance, Dataflow/Beam on Google platform).

My 2018 in review

My main objective in 2018 was to go deep in Machine Learning (as a way to continue 2017’s focus). When the year started, I decided to organize my free time in small 1-month projects. The original idea was to start with Deep Learning too, but I ended up exploring other fields, like data engineering.

In February I tried different approaches to develop a ML model to solve the famous Titanic Kaggle competition, where you have to predict the survival of different passengers given some data. It was really fun, because I explored different ideas, but I ended with a quite over-engineered notebook. Later I realized how important is to find which are the noisy features that you have to ignore.

In March I decided to improve my python skills so I set a challenge based in intuition: try to group products that are bought together, using real data from work (Ulabox, an online supermarket). I enjoyed creating sparse matrices with scipy and doing matrix operations with numpy, which was a good refresh of maths. The result was a nice dentogram that showed that some vegetables are bought together, as well as some types of yogurt.

In May I created a simple notebook to solve the Titanic competition but with one idea: help my work mates to join a competition and get excited with ML. So I made the most simple code that worked, but at the same time trying to show an eye-catching result. I tried plotting a simple decision tree with a great result: both coworkers joined the session, and other Kaggle users voted up my notebook.

In June I bought a new computer with a GTX1080, getting ready to jump to Deep Learning. I tried some tutorials (Tensorflow and Pytorch), but I didn’t like starting from level 0, that is, creating my own neurons from scratch. Actually I learned about neural networks years ago, at university. Later, almost at the end of 2018 I finally found a book with the level I was looking for: Advanced Deep Learning with Keras.

Regarding conferences: in July I attended PyData Berlin thanks to my employee (who paid me the tickets). Later in September I also attended DataEngConf in Barcelona, that really matched the needs of my company: make a data engineering plan. In October I took a train to Paris and then another to Karlsruhe to attend PyConDE; this conference was really well organized, with a wide concepts’ talks and in an incredible venue: a digital art museum with thought-provoking expositions about the future we are building.

The most interesting books I’ve read this year came as suggestions from conferences’ sessions: one is Lean Analytics and other is Data Engineering Teams. During 2018 I read some non work related books too, most of them sci-fi novels (like The Expanse book 3 and 4).

During summer I continue improving my knowledge of Python, using libraries to create images and videos. Also joined a MOOC about Google Cloud Platform (as a need from work).

I sent 3 papers for different Call for Papers during the year, and was lucky to get selected to do a workshop in November in Barcelona, during the unforgettable PyDay. I prepared a practical introduction to NLP, using classic and modern methods to classify texts. I chose Spanish jokes as the corpus to work on, and the result was amazing: both the audience and myself enjoyed a lot the workshop.

Finally in December I took a rest regarding tech stuff… and got married 🙂