Sunday, 15 April 2012

ggplot2 Time Series Heatmaps

How do you easily get beautiful calendar heatmaps of time series in ggplot2? E.g:
From MarginTale
I was impressed by the lattice-based  implementation from Paul Bleicher of Humedica, which you can find referenced in http://blog.revolutionanalytics.com/2009/11/charting-time-series-as-calendar-heat-maps-in-r.html. Then, when other blogs like http://timelyportfolio.blogspot.com/2012/04/piggybacking-and-hopefully-publicizing.html picked up the topic, I decided to try a ggplot2 implementation. In a comment to the above Revolution Analytics post, Hadley already presented a quick ggplot rendition, upon which I build here.

How do you attack the problem? Looking at the example output above:
  1. We facet_grid by "months" and "years" 
  2. The data itself is plotted by "week of month" and "day of week" and coloured according to the value of interest
So, given a time series we just have to fiddle with time indexes to create a data.frame containing the time series as well as per observation the corresponding "month", "year", "week of month", "day of week". The rest is then a one-liner of code with Hadley's wonderful ggplot2 system.

The following code contains step by step comments:

require(quantmod)
require(ggplot2)
require(reshape2)
require(plyr)
require(scales)
# Download some Data, e.g. the CBOE VIX
getSymbols("^VIX",src="yahoo")
# Make a dataframe
dat<-data.frame(date=index(VIX),VIX)
# We will facet by year ~ month, and each subgraph will
# show week-of-month versus weekday
# the year is simple
dat$year<-as.numeric(as.POSIXlt(dat$date)$year+1900)
# the month too
dat$month<-as.numeric(as.POSIXlt(dat$date)$mon+1)
# but turn months into ordered facors to control the appearance/ordering in the presentation
dat$monthf<-factor(dat$month,levels=as.character(1:12),labels=c("Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"),ordered=TRUE)
# the day of week is again easily found
dat$weekday = as.POSIXlt(dat$date)$wday
# again turn into factors to control appearance/abbreviation and ordering
# I use the reverse function rev here to order the week top down in the graph
# you can cut it out to reverse week order
dat$weekdayf<-factor(dat$weekday,levels=rev(1:7),labels=rev(c("Mon","Tue","Wed","Thu","Fri","Sat","Sun")),ordered=TRUE)
# the monthweek part is a bit trickier
# first a factor which cuts the data into month chunks
dat$yearmonth<-as.yearmon(dat$date)
dat$yearmonthf<-factor(dat$yearmonth)
# then find the "week of year" for each day
dat$week <- as.numeric(format(dat$date,"%W"))
# and now for each monthblock we normalize the week to start at 1
dat<-ddply(dat,.(yearmonthf),transform,monthweek=1+week-min(week))
# Now for the plot
P<- ggplot(dat, aes(monthweek, weekdayf, fill = VIX.Close)) +
geom_tile(colour = "white") + facet_grid(year~monthf) + scale_fill_gradient(low="red", high="yellow") +
opts(title = "Time-Series Calendar Heatmap") + xlab("Week of Month") + ylab("")
P

It should be easy to wrap into a function and I hope its useful.

Sunday, 4 March 2012

Boxplots and Day of Week Effects

THIS BLOG DOES NOT CONSTITUTE INVESTMENT ADVICE. ACTING ON IT WILL MOST LIKELY BE DETRIMENTAL TO YOUR FINANCIAL HEALTH.

After following some R-related quant finance blogs like Timely PortfolioSystematic Investor or Quantitative thoughts-  to name some of my favourites - I decided to start my own. I'll first focus on R snippets which come in handy, and will potentially expand to quant trading and backtesting as time allows.

I'll start with a simple graphical boxplot analysis of "days of the week effects" with two R snippet/tidbits regarding:
  1. How do you adapt the ggplot2 plotting of boxplots to a mundane 50%-box 95%-line 5%-dots view?
  2. How do you subdivide your days in weekdays easily and robustly? 

Lets jump directly into the code which can be downloaded at https://gist.github.com/1974563:


require(quantmod)
require(ggplot2)
require(reshape2)
# The standard definitions of boxplots are non-obvious to interpret for non-statisticians.
# A "the box is fifty percent, the line 95% and there you have 5% outlier points" is
# typically more easily swallowed by practitioners.
# I therefore define two functions which will change the boxplot appearance below.
myBoxPlotSummary <- function(x) {
r <- quantile(x, probs = c(0.025, 0.25, 0.5, 0.75, 0.975),na.rm=TRUE)
names(r) <- c("ymin", "lower", "middle", "upper", "ymax")
r
}
myBoxPlotOutliers <- function(x) {
tmp<-quantile(x,probs=c(.025,.975),na.rm=TRUE)
subset(x, x < tmp[1] | tmp[2] < x)
}
# Download some Data, e.g. the CBOE VIX
getSymbols("^VIX",src="yahoo")
# Make a factor depending on the day of week. We will use this to segement data according to days of the week.
wd<-factor(.indexwday(VIX),levels=1:5,labels=c("Mon","Tue","Wed","Thu","Fri"),ordered=TRUE)
# Note here that I do not use the weekdays function, because this will be locale dependent and lead
# to an unwanted sorting of the days in the boxplot
wd<-factor(.indexwday(VIX),levels=1:5,labels=c("Mon","Tue","Wed","Thu","Fri"),ordered=TRUE)
# wd<-factor(.indexwday(VIX),levels=1:7,labels=c("Mon","Tue","Wed","Thu","Fri","Sat","Sun"),ordered=TRUE)
# a dataframe with the factor and the daily returns from close to close
tail(mydf<-data.frame(wd=wd,ROC(Cl(VIX))))
mdat<- melt(mydf)
# plot the boxplots with own summary functions and outliers
ggplot(mdat,aes(wd,value)) +
opts(title = "Daily returns of the VIX") + xlab("") + ylab("% per day") +
stat_summary(fun.data=myBoxPlotSummary, geom="boxplot") +
stat_summary(fun.y = myBoxPlotOutliers, geom="point")
#kruskal.test(x=mydf[,2],g=mydf[,1])
Running the code, we get following output:
From MarginTale


These boxplots now show 50% of the observations in the box, the vertical lines cover 95% and the dots 2.5%. I find this easier to communicate than the standard definition. This is implemented in the functions myBoxPlotSummary and myBoxPlotOutliers which are in turn called from stat_summary in ggplot.

A second issue I tripped over is the sorting of days in the above boxplot. If one uses the obvious way and just defines a factor as "weekdays(index(...))" then the plot function will alphabetically sort the days - not exactly what you want. If you then try to order the factors, your solution will depend on how locale (the language you use) specifies the abbreviations of the weekdays. A robust solution shown  in the code is to use the function .indexwday from the package xts.