A stock-market index of small caps is at its weakest versus the S&P 500 since the financial crisis, suggesting a liquidity problem is brewing.