EDA
I work with data a lot, but the nature of my job isn't to dive super deep into a small amount of datasets, I'm often jumping between several projects every day and need to just get a super quick glance at some tables to get a high level view.
When I'm doing more interactive exploration I've graduated from Jupyter cells with df_N.head() to using an amazing tool called visidata
However, Visidata is a terminal based application and I'm often in an iPython console... so is there a way to move even faster for my super quick summary views?
yes!
Skimpy
First thing to do is pip install skimpy and then it's as easy to get some summary stats with skimpy <data>
This is super nice for seeing missing values in particular as well as the distribution shape of the data.
iPython
But wait... I just said I'm normally in an iPython session but that was called from zsh.. If I'm hoping back into zsh I might as well use visidata to have more powerful exploration at my fingertips. So... can I see this table quickly without breaking my iPython workflow?
Of course you can with magic!
The above assumes you're looking at a file, like you would in the terminal.
skimpy works even better in iPython with from skimpy import skim then pass any DataFrame to skim!
