If the tmp space grew too large and had to resize then we would have stale pointers. Keep around the offset instead so we don't store invalid pointers.