Extending the central concept of recurrence times for a point process to recurrent events in space-time allows us to characterize seismicity as a record breaking process using only spatiotemporal relations among events. Linking record breaking events with edges between nodes in a graph generates a complex dynamical network isolated from any length, time or magnitude scales set by the observer. For Southern California, the network of recurrences reveals new statistical features of seismicity with robust scaling laws. The rupture length and its scaling with magnitude emerges as a generic measure for distance between recurrent events. Further, the relative separations for subsequent records in space (or time) form a hierarchy with unexpected scaling properties.