Originally published on Treasure Data blog.
Magic functions, a mainstay of pandas, enable common tasks by saving you typing. Magic functions are functions preceeded by a % symbol. Magic functions have been introduced into pandas-td version 0.8.0! Toru Takahashi from Treasure Data walks us through.
Treasure Data’s magic functions work by wrapping a separate % around the original function, making the functions callable by %%. Letâs explore further to see how this works.
We start by creating a connection, importing our relevant libraries, and issuing a basic query, all from python (in Jupyter). Using the sample data, it would look like this:
import os import pandas_td as td #Initialize connection con = td.connect(apikey=os.environ[âTD_API_KEYâ], endpoint = âhttps://api.treasuredata.comâ) engine = con.query_engine(database=âsample_datasetsâ, type=âprestoâ) #Read Treasure Data query into a DataFrame df = td.read_td(âselect * from www_access, engineâ)
With the magic function
We can now do merely this:
%%td_use_sample_datasets %%td_presto select count(1) as cnt from nasdaq
If you add the table name nasdaq after %% td_use, you can also see the schema:
Even better, you can tab edit the stored column names:
As long as %matplotlib inline is enabled; then you can throw a query into magicâs %%td_presto – -plot and immediately visualize it!
How to enable it
Set the API_KEY environment variable:
You can then load the magic comment automatically! Youâll want to save the following to ~/.ipython/profile_default/ipython_config.py
c = get_config() c.InteractiveShellApp.extensions=[ âpandas_td.ipythonâ, ]
Stay tuned for many more useful functions from pandas-td! These tools, including Pandas itself, as well as Python and Jupyter are always changing, so please let us know if anything is working differently than whatâs shown here.