Advanced Python Regex: Groups, Lookarounds and Substitution (Part 2)

Advanced Python regex means named groups that pull fields out of text, lookarounds that check context without capturing it, and re.sub with a function that transforms every match. Together they turn the re module into a small parser. They also bring a trap: the innocent-looking pattern (a+)+$ took 10.7 seconds to reject a 24-character string in the timing measured below.

This part is for readers who know the basics from Part 1 on regular expressions in Python: raw strings, flags, and match versus search. You will be able to parse log lines into dictionaries, match numbers only in the right context, rewrite dates and redact card numbers, and spot a pattern that can hang your program before it ships.

How do capturing groups work?

Parentheses capture the text they match, and you retrieve it by number with group() or all at once with groups():

import re

log = "2026-07-29 14:32:05 ERROR Database connection failed"

m = re.search(r"(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2}) (\w+) (.+)", log)
m.group(1)    # '2026-07-29'
m.group(3)    # 'ERROR'
m.groups()    # ('2026-07-29', '14:32:05', 'ERROR', 'Database connection failed')

However, positional groups break the moment you insert a group in the middle. Named groups do not:

pattern = re.compile(
    r"(?P<date>\d{4}-\d{2}-\d{2}) "
    r"(?P<time>\d{2}:\d{2}:\d{2}) "
    r"(?P<level>\w+) "
    r"(?P<message>.+)"
)

m = pattern.search(log)
m.group("level")      # 'ERROR'
m["level"]            # 'ERROR' - same thing, shorter
m.groupdict()
# {'date': '2026-07-29', 'time': '14:32:05', 'level': 'ERROR',
#  'message': 'Database connection failed'}

groupdict() returns a ready-made dictionary, which is exactly what you want when parsing structured logs into records.

When should you use a non-capturing group?

Use (?:...) whenever you need grouping for alternation or a quantifier but do not need the text back. It keeps group numbers meaningful and stops findall changing what it returns:

# capturing - findall returns just the group, which surprises people
re.findall(r"(https?)://\S+", "see http://a.com and https://b.com")
# ['http', 'https']

# non-capturing - findall returns the whole match
re.findall(r"(?:https?)://\S+", "see http://a.com and https://b.com")
# ['http://a.com', 'https://b.com']

That findall behaviour catches everyone once. The re.findall documentation spells it out: no groups returns whole matches, one group returns that group, and several groups return tuples:

re.findall(r"(\w+)=(\d+)", "width=20 height=10")
# [('width', '20'), ('height', '10')]

Advanced Python regex syntax: quick reference

How do lookahead and lookbehind work?

Lookarounds are zero-width assertions: they check a condition without consuming characters. The Regular Expression HOWTO section on lookahead assertions walks through the idea in more depth.

Extracting a number only when it is a price:

text = "Costs 500 rupees, weighs 500 grams"

re.findall(r"\d+(?= rupees)", text)     # ['500'] - only the price

The " rupees" is checked but not included in the match.

Lookbehind for currency prefixes:

re.findall(r"(?<=₹)\d+", "₹1500 and 2000 units")    # ['1500']

Negative lookahead to exclude something, here the plural "cats" but not longer words that start with "cat":

words = "cat cats category catalogue"
re.findall(r"\bcat(?!s\b)\w*", words)     # ['cat', 'category', 'catalogue']

Lookarounds can also match a position with no characters at all. For example, this inserts thousands separators:

re.sub(r"(?<=\d)(?=(\d{3})+$)", ",", "1234567")    # '1,234,567'

Python's fixed-width lookbehind rule

Python's lookbehind must be fixed width. Both (?<=\d{2,4}) and (?<=Rs\.|INR ) raise re.error: look-behind requires fixed-width pattern, the second because its two alternatives have different lengths. Work around it with a group:

re.findall(r"(?:Rs\.|INR) ?(\d+)", "Rs.1500, INR 2000 and 300 units")
# ['1500', '2000']

How does substitution with re.sub work?

re.sub(pattern, replacement, text) replaces every match, and the replacement can be a string with backreferences or a function. That is where regex becomes a transformation tool:

re.sub(r"\s+", " ", "too    much   space")        # 'too much space'
re.sub(r"[^\w\s]", "", "Hello, World!")           # 'Hello World'

Backreferences let you reorder:

# DD-MM-YYYY to YYYY-MM-DD
re.sub(r"(\d{2})-(\d{2})-(\d{4})", r"\3-\2-\1", "29-07-2026")
# '2026-07-29'

# with named groups
re.sub(r"(?P<d>\d{2})-(?P<m>\d{2})-(?P<y>\d{4})",
       r"\g<y>-\g<m>-\g<d>", "29-07-2026")
# '2026-07-29'

The replacement is also a raw string, r"\3-\2-\1", for the same reason patterns are.

Using a function as the replacement

This is the feature worth knowing:

def redact(match):
    number = match.group(0)
    return number[:2] + "*" * (len(number) - 4) + number[-2:]

text = "Card 4532015112830366 used"
re.sub(r"\b\d{16}\b", redact, text)
# 'Card 45************66 used'

Each match is passed as a match object, and whatever you return is substituted. As a result, re.sub becomes arbitrary text transformation.

Another common use is normalising units:

def to_upper_unit(m):
    return f"{m.group('num')} {m.group('unit').upper()}"

re.sub(r"(?P<num>\d+)\s*(?P<unit>kb|mb|gb)", to_upper_unit, "5 mb and 2gb")
# '5 MB and 2 GB'

re.subn returns the count as well, which is useful when you need to know whether anything changed:

result, n = re.subn(r"\d+", "#", "a1b22c333")
# ('a#b#c#', 3)

How do you keep the delimiters when splitting with re.split?

Put the delimiter pattern in a capturing group. re.split then returns the separators in the result list alongside the pieces:

re.split(r"[,;]", "a,b;c")           # ['a', 'b', 'c']
re.split(r"([,;])", "a,b;c")         # ['a', ',', 'b', ';', 'c']

That is handy when you need to reconstruct the string later with the separators intact.

What is catastrophic backtracking, and how do you avoid it?

Catastrophic backtracking is exponential slowdown when nested quantifiers give the engine too many ways to split a failing input. Avoid it by removing the nesting, or by using an atomic group or possessive quantifier on Python 3.11 and later.

# dangerous
pattern = r"(a+)+$"

Against a run of a followed by X, the engine must try exponentially many ways to divide the a characters between the inner and outer quantifier before concluding failure. Measured on Python 3.14 on one Windows laptop, 18 characters took 0.18 seconds, 20 took 0.64 seconds, 22 took 2.8 seconds and 24 took 10.7 seconds. In other words, each extra pair of characters roughly quadrupled the time, so a few more characters take longer than you will wait.

The problem is nested quantifiers over overlapping character sets. The fix is to restructure rather than to micro-optimise:

pattern = r"a+$"          # the same strings, no nested ambiguity

Since Python 3.11 you can also stop the engine backtracking into a group. An atomic group (?>a+)+$ or a possessive quantifier (a++)+$ fails on the same 30-character input in well under a millisecond. Both are listed in the re module syntax reference.

More realistic examples show up in patterns like (\w+\s*)+ or (.*,)*. If a pattern has a quantifier inside a group that is itself quantified, look closely.

Defensive habits: prefer specific character classes to ., avoid nesting quantifiers, and test patterns against long non-matching input rather than only against strings that match.

A worked example: parsing web server logs

Pulling structured records out of semi-structured text brings these advanced Python regex features together:

log_line = re.compile(r"""
    ^(?P<ip>\d+\.\d+\.\d+\.\d+)\s+
    -\s+-\s+
    \[(?P<ts>[^\]]+)\]\s+
    "(?P<method>[A-Z]+)\s(?P<path>\S+)[^"]*"\s+
    (?P<status>\d{3})\s+
    (?P<size>\d+|-)
""", re.VERBOSE)

lines = [
    '203.0.113.7 - - [29/Jul/2026:14:32:05 +0530] "GET /index.html HTTP/1.1" 200 5120',
    '198.51.100.23 - - [29/Jul/2026:14:32:09 +0530] "POST /api/login HTTP/1.1" 401 -',
    "garbage line",
]

records = []
for line in lines:
    m = log_line.match(line)
    if m:
        records.append(m.groupdict())

for r in records:
    print(r)

Output:

{'ip': '203.0.113.7', 'ts': '29/Jul/2026:14:32:05 +0530', 'method': 'GET', 'path': '/index.html', 'status': '200', 'size': '5120'}
{'ip': '198.51.100.23', 'ts': '29/Jul/2026:14:32:09 +0530', 'method': 'POST', 'path': '/api/login', 'status': '401', 'size': '-'}

The garbage line is skipped silently because match returns None. Note [^\]]+ for the timestamp rather than .+?: a negated class cannot overshoot the closing bracket, so it is both clearer and less prone to backtracking. Every value comes back as a string, so convert status and size yourself.

Common mistakes with advanced Python regex

• Adding a group and breaking findall. A single capturing group changes the return value from whole matches to group contents. Use (?:...) when you only need grouping.
• Variable-width lookbehind. It raises re.error; restructure with a capturing group.
• Forgetting the r prefix on replacements. "\1" is a control character, not a backreference.
• Passing count or flags positionally to re.sub. It is deprecated since Python 3.13 and easy to get wrong; use keywords.
• Testing only strings that match. Backtracking problems appear on long inputs that fail.

Related reading

Regex is not always the answer: if str.split, startswith or an in check does the job, it is easier to read and test. Mastering Python strings shows those methods. Regular expressions in Python, Part 1 covers the fundamentals. For processing large log files one line at a time, see iterators and generators in Python.