The mechanical behavior of granular materials is widely governed by microstructure reorganizations. This constant evolution of the microscale geometry is often taken into account directly at the macroscale within the plasticity context, and more particularly non-associated plasticity. In this paper we propose to revisit the non-associated plastic behavior of granular materials with respect to material instability assessment in terms of the loss of positiveness of the second-order work. It is shown that large incremental plastic strain is a necessary condition for the existence of mechanical instability. The ability of a material to develop substantial plastic incremental strain is then related to the existence of rattlers, i.e., free particles that get jammed into force chains when existing contact networks fail to adapt to incremental loadings. This link between rattlers, plastic deformation and the vanishing of the second-order work is explicitly derived i) from the micro-formulation of the second-order work and ii) directly from discrete element simulations.