When Gradients Explode (or Vanish) 💥

on

This is a nice quick read on how to combat exploding or vanishing gradients, a problem that wreak havoc on your deep learning model. 👹

My TL;DR:

  • Exploding gradient? Use gradient clipping. It sets a ceiling on your gradient but keeps the direction. ✂️
  • Vanishing gradient? If you’re using an RNN, use an LSTM. ✔️

Src: Learn.Love.AI.