Make _parse_imro_string self-contained by deriving IMRO field schemas from the catalogue

[h-mayorquin]

This is another PR in the series of Neuropixels cleanups (see #410, #405) with the goal of making everything come from a single source of data: the Neuropixels probe library from ProbeTable.

The following victim is _parse_imro_string, which required callers to pass a probe_part_number just so it could look up the IMRO field schema, even though the IMRO string itself contains the format type code needed to determine the schema. This forced us to maintain a hand-written dict (probe_part_number_to_probe_type) that duplicated information already present in the ProbeTable catalogue, and read_imro had to pre-parse the header and reverse-lookup through it. The dict had to be manually updated whenever a new probe was added, and it had already drifted from the catalogue (missing types like 1120, 1122, 1123, 1200, 3010, 3020).

I added a post-processing script (resources/postprocess_neuropixels_probe_features.py) that derives two mappings from the catalogue JSON after each weekly ProbeTable sync: z_imro_format_type_to_imro_format (format type code to field schema) and z_imro_format_type_to_part_number (format type code to canonical part number for geometry construction). _parse_imro_string now takes just the IMRO string, extracts the format type from the header, and looks up the schema directly. read_imro uses the part number mapping instead of the hand-maintained dict.

Something to discuss: while working on this, I noticed that some format type codes in the old dict (NHP types 1015, 1021, 1022, 1031, 1032 and NP2.x variants 2004, 2014) are not listed in any ProbeTable val_def entry. My recollection is that the old dict had them because it was originally designed to map part numbers to types (where one entry per part number made sense) and was later repurposed in reverse to resolve types back to part numbers. I have left them in a small fallback dict for backwards compatibility, but I think we should remove them after a release or two if nobody reports a breakage. What do you think?

[samuelgarcia]

Thank a lot for the deep and long effort to clean up total the messy zone.

[chrishalcrow]

Here's some real test data from Jennifer Colonell, if you'd like it.
test_np2020.txt

Looks good!

[alejoe91]

I think these should be added to the ProbeTable val def. But let's ask Bill: billkarsh/ProbeTable#5

[alejoe91]

We can remove these as they are not valid probe types: billkarsh/ProbeTable#5 (comment)

Probe types are categories, not part numbers.

[alejoe91]

Thanks @h-mayorquin

Let's wait to see if it makes sense to add them to ProbeTable, or if we can safely remove them because they are not officially supported codes for probe types: billkarsh/ProbeTable#5

[alejoe91]

Suggested change

	# IMRO type codes not listed in any val_def entry in the ProbeTable catalogue.
	# These probes all use the imro_np1000 or imro_np2003/imro_np2013 format, but their
	# type codes are not in the corresponding val_def type sets.
	# We don't know if SpikeGLX actually produces IMRO files with these type codes
	# (there is no test data for them). They are kept here for backwards compatibility.
	# Values are (imro_format_name, canonical_part_number).
	#
	# TODO: @team - Should these be added to ProbeTable's val_def, or can they be removed?
	# If SpikeGLX never produces these type codes, this dict can be deleted entirely.
	_imro_format_type_fallback = {
	"1015": ("imro_np1000", "NP1015"),
	"1021": ("imro_np1000", "NP1021"),
	"1022": ("imro_np1000", "NP1022"),
	"1031": ("imro_np1000", "NP1031"),
	"1032": ("imro_np1000", "NP1032"),
	"2004": ("imro_np2003", "NP2004"),
	"2014": ("imro_np2013", "NP2014"),
	}

See billkarsh/ProbeTable#5 (comment)

[h-mayorquin]

Ok, removed it

[alejoe91]

Suggested change

	elif first_value in _imro_format_type_fallback:
	imro_format_type = first_value
	imro_format = _imro_format_type_fallback[imro_format_type][0]

[h-mayorquin]

Ok, removed it.