Neither here nor there means to be irrelevant to the subject.
1. So what if Paritosh comes late? That is neither here nor there; the meeting will still be scheduled at the fixed time.
2. Making such a volatile statement was neither here nor there; it would not help the cause at all.