Above mentioned was for version 3.61 of the plugin. I have updated to the latest 3.63 and the problem persists.
WP version is still 3.6.1.
To exclude that the combination of this plugin with my theme is what makes the problem, I've switched temporarily to the default Twenty Thirteen theme and it's the same. 1s load time with plugin disabled vs. 5s load time with the plugin enabled, for each and every page.