I've noticed my Vim got horribly slow once I had multiple panes in tmux open.
My initial investigation led to the fact, that tmux is slowing Vim down, and a quick Google search confirmed that.
After removing all plugins in Vim/tmux, I dug through the iTerm2 settings and found this:
Removing the tick at "Save lines to scrollback in alternate screen mode" and setting the scrollback to a reasonable amount (I chose 1000), the lag was nearly gone. I'll continue investigating, to improve the Vim performance.