Transformers have completely taken by storm the field of sequence modelling with deep networks, becoming the standard for text processing, video, even images. RNNs that were once a very active engineering field have slowly faded into the void. All of them? No, some RNNs are bravely fighting back to claim state-of-the-art results in sequence tasks. The most suprising part?

This post is an attempt at summarizing results obtained over five years of research in the laboratory of Srdjan Ostojic, at ENS Paris, on the uses of low-rank RNNs, in a ten-minutes read. Let’s see if that’s enough to catch your interest! This post has been written with invaluable inputs from several lab members, in particular Friedrich Schuessler, Francesca Mastrogiuseppe and Srdjan Ostojic. Many thanks to them! Why low-rank RNNs?

Between self-supervised approaches for transfer learning, contrastive losses and representational similarity analysis, these last few years were as rich in ideas as buzzing with confusing words. Here is a little dictionary to celebrate the end of 2021. If you have time for only one entry, check out the self-supervised learning one above all!

When I try to avoid working on my doctoral projects, I like to get lost in the statistical world, to marvel at the seemingly infinite amount of statistical methods out there, and think how they are all related and yet different. There is one particular tool that I have recently seen in many different studies in neuroscience and has caught my attention: Canonical Correlation Analysis, or CCA for friends.

Linear regression is to statistics what Wonderwall is to guitar : you learn it at the very beginning, it looks simple and naive, and after you saw it once you don’t really want to hear about it again. But surprisingly it hides a lot of subtleties that I keep discovering years about seeing my first linear regression.

As humans have been grappling with a pretty rough ride this year, how have machines been doing? Let’s find out in this small review! Welcome, Gigantic Prolific Transformer III Let’s start with one of the most remarkable and buzz-generating AI releases of the year, I have named the infamous GPT-3, latest language model of a series developed by OpenAI.

Having machines with many CPU cores is a great way to make all your computations run much faster, right?

As I come from the computer science community, I find that git is a very useful tool for handling projects, although it can be intimidating at first. This is my attempt at get you going on git in a few minutes. Of course, if you want to really master it, nothing beats the official tutorial, but it is maybe too much for a basic usage.