Regex Resources for Catalogers and Others in the Library

2024-11-22 cataloging, metadata, tech, library

After last week’s talk on making use of the period between now and fundamental shifts in cataloging, regular expressions (regex) came up during the Q&A as one of the things I think it’s helpful for catalogers to learn. Afterward, one of the attendees emailed me asking for more specific recommendations on how to learn them. I’m sharing the resources I pulled together, since I think they could be of use to more folks. I would also recommend that, if you get the opportunity, this is a good one to learn in a workshop where you can ask the instructor questions.

What are Regular Expressions?

And what are they good for?

Regular expressions are search patterns you can use to match more than a single thing. You can use them to search for anything from minor variations of the same search term to patterns throughout a file. You can use them to find things but also to manipulate them, shuffling data into new patterns. There may be slight variations between systems because they can be implemented in different kinds of software, but they’re generally easy to reuse across systems and programs. Here are some examples of what they can do:

A super simple regular expression would be: gr[ae]y. You’d use this to search for either gray or grey. The [] specifies a “character class” or a group of characters. It will look for matches to any of the characters.

You can also search for things where data may or may not be present, e.g. https?:\/\/. The ? means that it’s search for 0 or 1 instance of s. So this will match http:// and https:// (the // are special characters and had to be “escaped” with a \).

You can use shortcodes to represent certain things and parentheticals to identify groupings to return. For example, when I was doing a giant update on call numbers to change HCLA 1421 box03 AX/SP/10019/06 to 01421 box03 AX/SP/10019/06 I could search for HCLA (\d{4}) and replace it with 0$1 . This search looks for call numbers where there were four digits (\d) (followed by a space) and putting one leading 0 in front of them (and not putting in the HCLA because we weren’t putting that data in call numbers any more). You can repeat this for call numbers with only one digit, two digits, three digits, etc., and put in the correct number of leading 0s so they’re all 5-digits long.

And once you’ve leveled up, you can search for any kind of pattern, e.g. ^(.+?)\t(.+?)\t(.+?)$ will search for three units of tab-separated data. Using references to these three parentheticals (as $1 $2 and $3) in a replace field would let me change their order or otherwise recreate them. I’ve used this kind of thing to clean up or revise whole sheets of metadata.

Resources

So, as I said above, I would recommend that if you have a chance to learn regex in some kind of interactive course or workshop, it’s great to be able to talk through this kind of thing with an instructor. But, fortunately, there are plenty of resources online to let you experiment.

Regex Sandbox

First, where can you use regular expressions? They’re built into most fancier text editors (Notepad++, Sublime Text, VSCode) although I find the design of Notepad++’s find/replace overall to be frustrating and wouldn’t suggest starting there. But there are also sandboxes built specifically for working with regular expressions. I recommend Regexr.

Don’t worry about the settings at first. Use the top line, Expression to build your search string. Put the Text you want to search in the next box down (ignore the Tests toggle for now!). Then in the Tools box you can do a few things… you can put in a second function to Replace something, have it List all the occurences it found, look at Details (Which I never use), or have it Explain the search string you’ve written.

Regexr is good for practice but it can also be handy in your day-to-day. Sometimes if I’m doing a quick find/replace on a limited amount of data, I just paste the original in here, run the find/replace, and copy out the result! It can be faster than opening a text editor.

Here’s examples of Replace and Explain for the call number example I gave above:

Screenshot of Regexr showing an upper box with the HCLA search example given above, the text box with the sample call and box number, and the replace shortcode, followed by the outcome Replace: functionally it’s removed the HCLA and offset the selected number to five digits

$Screenshot showing the explanation: Parentheticals are capturing group #1. Groups multiple tokens together and creates a capture group for extracting a substring or using a backreference. Then it shows the \d is Digit. Matches any digit character (0-9). The {4} is a Quantifier. Match 4 of the preceding token.$ Explain: It provides a nested explanation of number select portion

You don’t have to understand it all up front, that’s what the courses are for!

Regex Courses

First, Library Carpentry has an online version of their Regex course. If you read the text in full and review the questions and answers, it’s a pretty decent introduction. It’s also something you can keep open and revisit while watching the next video.

After reviewing some YouTube videos, I found a pretty good Regex tutorial. Here, you actually get to see the search queries in action and the teacher explains things as he goes. It may move a bit fast for someone who doesn’t already have background, which is one of the reasons I think it’s a good idea to start with the Library Carpentry course. You can also mitigate it by pausing frequently, rewatching, and then trying the examples yourself over on Regexr.

As a bonus, check out this post on regex by Bohyun Kim. She includes some examples of how it might be relevant in libraries + some fun leveling-up stuff like lookaheads. You may notice that she uses \1 or \2 for references where I used $1, etc. That’s an example of where there can be minor differences between programs. Some accept both!

My Own Regex Notes

Finally, these are my own reference notes for regular expressions. Feel free to copy and save for yourself. I’ve also put in some MarcEdit specific ones at the end, in case they’re helpful once you’ve gotten a handle on them – but I wouldn’t start with trying to figure those out. See also the Mozilla Developer Network regex cheatsheet.

Character Classes

Character classes are enclosed by [ ]. Put a ^ at the beginning to make it a negative match.

[aqT,] will match the first of any of those 4 characters you hit and ONLY that.
[a-c4z6-8] matches first instance of a, b, c, 4, z, 6, 7, 8
[^a] matches anything except an “a” or [^a-c] will match anything but a, b, c
^[0-9]+ matches entire integer at the beginning of the line/string (note the ^ and the +, see below)
^[a-zA-Z0-9]+ matches an alphanumeric string

Shorthand Character Classes

There are also shortcuts when you’re trying to match anything or anything EXCEPT something.

\w word-character (letters AND numbers) & \W everything else (spaces, symbols, etc.)
\d = digit & \D = everything else (letters, symbols, spaces)
\s = whitespace & \S = everything else
\t = tab
\r = return & \n = new line (sometimes you have to combine them like \r\n)

Quantifiers

Specify more about where to search and how many of a thing to search.

^ — match string at the beginning of a line
- ^PHP will match “PHP” at the very beginning of a line but not midway through a line
$ — match string at the end of a line
- pdf$ will only match the letters “pdf” if they’re the last thing in the line.
. — single instance of anything
+ — one or more of whatever it follows
* — zero or more of whatever it follows
? — zero or one of whatever it follows
- e.g. https? matches http or https but wouldn’t match all of httpss
* and + are greedy, so they will keep going as far as they can until they hit the last instance of a thing
? — kills greed
- so P.*?P will match anything until it hits the first P. So PIP, PHP, PHHHHHP, but if it encountered PIPE PLAYER, it would only match the PIP whereas P.*P would match PIPE P.

Special Characters

Quantifiers above are special characters, but also: \ and / themselves are special characters.

Escaping special characters — \. = . (an actual period) or ^\$ would match a string beginning with $.

Subpatterns

Subpatterns allow you to make “or” statements and/or to find a specific part of the pattern as a separately-identified pattern. This can be helpful later for backreferences.

subpatterns are enclosed in ( )
(cat) = cat and returns cat, (cat)+ = catcatcat to match but returns only “cat” (at least in some forms)
(cat|dog) = cat or dog and returns whichever it matches
HCLA (\d{4}) = matches the HCLA and four numbers but only selects the four numbers as a subpattern
((https?|ftp|gopher|file):\/\/(.+?))\s gets domains (if they’re followed by a space) and also selects the prefix and the rest of the URL as additional subpatterns (see below).

Back References

Identifying stuff is good, but then how do we reference it once we’ve found and identified it? Some systems use $1 and some use \1.

(outer(inner)) $1 = outer, $2 = (inner)
(sub1)(sub2) $1 = sub1, $2 = sub2
((https?|ftp|gopher|file):\/\/(.+?))\s = $1 is the whole thing, $2 is the http/s or ftp or gopher or file and $3 is the rest of the URL.

Regex for MarcEdit

From Terry’s quickref:

Substitution (backreference) syntax: uses $ vs. \
Match all of a hundreds/tens group: Instead of 1xx, use 1\d{2}.
Field names: =500.{4} catches field name, spaces, and indicators
Field names with contents: =041.{4}\$a\w{3}$: 041 field with only a single language code in it
Handling $: [^$]* - searches for anything which is NOT a $. aka strings grabs up to the $. You don’t have to escape the $.
Match all captured: $0 - matches whole set of captured things.
Additionally: lcase() & ucase() when returning backreferences - for matching [A-Z] or [a-z] and \W to match non-word values. For example, ([A-Z])([A-Z\W]*) matches anything capitalized followed by all caps/non-word characters. Then replace as $1lcase($2) so first letter stays capitalized.