The mechanical behavior of amorphous solids such as glasses, gels and granular materials is still not well understood. One important aspect of this behavior is the transition under external forces, from rigid, elastic behavior to flowing, plastic response. The point at which this occurs may be expected to be a non-equilibrium critical point. We have found that the point of transition is not a critical point in the usual sense but a phenomenon known in the dynamical systems literature as a ``transition to chaos''. Similarly to a critical point, the material gains ergodicity at this point, but the transition is not associated with a diverging correlation length. A critical point with a diverging correlation length does occur in these materials but at a different level of forcing.