Coding is a rollercoaster of efficiency and eyebrow-raising discoveries.

Data Engineers or Developers - many of us love to be gourmet chefs in the kitchen. When it comes to planning and design, we would rather throw all ingredients in the pot and see how it comes out. 

Coding without a plan is like assembling a puzzle in a dark room. The result will most probably be unexpected and off the canvas.

Whether you follow Waterfall or Agile development strategy, planning and design phases are non-negotiable and are essential to reduce development cycles and redo work.

Once upon a time, one data engineer created an amazing piece-of-cake automation pipeline. This masterpiece had very complex logic, pulling data from multiple sources, and merging and persisting the data in a complex, incremental way. When the pipeline started to run successfully and automation flows worked, the data engineer got very excited and considered this development done. A few days later QA engineer found out that the result dataset was never created in the destination. Why has that happened? The excitement of the flow features and deployment success were strong. Target table existence was not listed in the success metrics? Or maybe there were no success metrics planned and listed at all?

When we consider some task done, neurons in our head are doing a happy dance stimulated by a surge of the hormone dopamine.  Therefore, the 80/20 rule makes us happier.  We dash to a minimal viable success with 80% of work done, get dopamine "high-fives" from our brain; and rush to the next task and the next dose of happiness. Working on the leftover 20% is boring and can take ten times more than the first 80%.

When you clean your room, pick up 80% of the mess and spend the remaining part of the day wondering how on earth a sock ended up in the fridge. Code-wise, it's a rollercoaster of efficiency and eyebrow-raising discoveries.