# Advanced Python Regex: Groups, Lookarounds and Substitution (Part 2)

> Advanced Python regex with tested output: named groups, lookarounds, re.sub with a function, and why a 24-character input made (a+)+$ run for 10 seconds.

- **Author:** Ayushi Kulshreshta — AI Engineer (https://www.1stepgrow.com/authors/ayushi-kulshreshta/)
- **Published:** Jul 22, 2026 · **Updated:** Sep 17, 2026
- **Topic:** Python & Programming · **Format:** Guide · **Read time:** 9 min
- **Canonical URL:** https://www.1stepgrow.com/articles/advanced-python-regex/

## Key takeaways

- Named groups make patterns self-documenting and survive reordering, and groupdict() turns a match into a ready-made dictionary.
- If a pattern contains capturing groups, re.findall returns the groups rather than the whole match.
- Lookarounds assert without consuming, which is how you match context without capturing it; Python's lookbehind must be fixed width.
- re.sub accepts a function as the replacement, which turns substitution into arbitrary transformation.
- Nested quantifiers over overlapping patterns can backtrack catastrophically, so restructure the pattern or use an atomic group.

Advanced Python regex means named groups that pull fields out of text, lookarounds that check context without capturing it, and `re.sub` with a function that transforms every match. Together they turn the `re` module into a small parser. They also bring a trap: the innocent-looking pattern `(a+)+$` took 10.7 seconds to reject a 24-character string in the timing measured below.

This part is for readers who know the basics from [Part 1 on regular expressions in Python](https://www.1stepgrow.com/articles/regular-expressions-in-python): raw strings, flags, and `match` versus `search`. You will be able to parse log lines into dictionaries, match numbers only in the right context, rewrite dates and redact card numbers, and spot a pattern that can hang your program before it ships.

## How do capturing groups work?

Parentheses capture the text they match, and you retrieve it by number with `group()` or all at once with `groups()`:

```python
import re

log = "2026-07-29 14:32:05 ERROR Database connection failed"

m = re.search(r"(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2}) (\w+) (.+)", log)
m.group(1)    # '2026-07-29'
m.group(3)    # 'ERROR'
m.groups()    # ('2026-07-29', '14:32:05', 'ERROR', 'Database connection failed')
```

However, positional groups break the moment you insert a group in the middle. Named groups do not:

```python
pattern = re.compile(
    r"(?P<date>\d{4}-\d{2}-\d{2}) "
    r"(?P<time>\d{2}:\d{2}:\d{2}) "
    r"(?P<level>\w+) "
    r"(?P<message>.+)"
)

m = pattern.search(log)
m.group("level")      # 'ERROR'
m["level"]            # 'ERROR' - same thing, shorter
m.groupdict()
# {'date': '2026-07-29', 'time': '14:32:05', 'level': 'ERROR',
#  'message': 'Database connection failed'}
```

`groupdict()` returns a ready-made dictionary, which is exactly what you want when parsing structured logs into records.

## When should you use a non-capturing group?

Use `(?:...)` whenever you need grouping for alternation or a quantifier but do not need the text back. It keeps group numbers meaningful and stops `findall` changing what it returns:

```python
# capturing - findall returns just the group, which surprises people
re.findall(r"(https?)://\S+", "see http://a.com and https://b.com")
# ['http', 'https']

# non-capturing - findall returns the whole match
re.findall(r"(?:https?)://\S+", "see http://a.com and https://b.com")
# ['http://a.com', 'https://b.com']
```

That `findall` behaviour catches everyone once. The [re.findall documentation](https://docs.python.org/3/library/re.html#re.findall) spells it out: no groups returns whole matches, one group returns that group, and several groups return tuples:

```python
re.findall(r"(\w+)=(\d+)", "width=20 height=10")
# [('width', '20'), ('height', '10')]
```

## Advanced Python regex syntax: quick reference

| Syntax | Name | What it does |
|---|---|---|
| `(...)` | Capturing group | Stores the match, numbered from 1 |
| `(?P<name>...)` | Named group | Stores the match under a name |
| `(?:...)` | Non-capturing group | Groups without storing |
| `(?=...)` | Positive lookahead | Followed by |
| `(?!...)` | Negative lookahead | Not followed by |
| `(?<=...)` | Positive lookbehind | Preceded by (fixed width only) |
| `(?<!...)` | Negative lookbehind | Not preceded by (fixed width only) |
| `\1`, `\g<name>` | Backreference | Refers to a group in a replacement |
| `(?>...)` | Atomic group (3.11+) | Never backtracks into the group |
| `a++`, `a*+` | Possessive quantifier (3.11+) | Like `+` and `*`, without backtracking |

## How do lookahead and lookbehind work?

Lookarounds are zero-width assertions: they check a condition without consuming characters. The [Regular Expression HOWTO section on lookahead assertions](https://docs.python.org/3/howto/regex.html#lookahead-assertions) walks through the idea in more depth.

Extracting a number only when it is a price:

```python
text = "Costs 500 rupees, weighs 500 grams"

re.findall(r"\d+(?= rupees)", text)     # ['500'] - only the price
```

The `" rupees"` is checked but not included in the match.

Lookbehind for currency prefixes:

```python
re.findall(r"(?<=₹)\d+", "₹1500 and 2000 units")    # ['1500']
```

Negative lookahead to exclude something, here the plural "cats" but not longer words that start with "cat":

```python
words = "cat cats category catalogue"
re.findall(r"\bcat(?!s\b)\w*", words)     # ['cat', 'category', 'catalogue']
```

Lookarounds can also match a position with no characters at all. For example, this inserts thousands separators:

```python
re.sub(r"(?<=\d)(?=(\d{3})+$)", ",", "1234567")    # '1,234,567'
```

### Python's fixed-width lookbehind rule

Python's lookbehind must be **fixed width**. Both `(?<=\d{2,4})` and `(?<=Rs\.|INR )` raise `re.error: look-behind requires fixed-width pattern`, the second because its two alternatives have different lengths. Work around it with a group:

```python
re.findall(r"(?:Rs\.|INR) ?(\d+)", "Rs.1500, INR 2000 and 300 units")
# ['1500', '2000']
```

## How does substitution with re.sub work?

`re.sub(pattern, replacement, text)` replaces every match, and the replacement can be a string with backreferences or a function. That is where regex becomes a transformation tool:

```python
re.sub(r"\s+", " ", "too    much   space")        # 'too much space'
re.sub(r"[^\w\s]", "", "Hello, World!")           # 'Hello World'
```

Backreferences let you reorder:

```python
# DD-MM-YYYY to YYYY-MM-DD
re.sub(r"(\d{2})-(\d{2})-(\d{4})", r"\3-\2-\1", "29-07-2026")
# '2026-07-29'

# with named groups
re.sub(r"(?P<d>\d{2})-(?P<m>\d{2})-(?P<y>\d{4})",
       r"\g<y>-\g<m>-\g<d>", "29-07-2026")
# '2026-07-29'
```

The replacement is also a raw string, `r"\3-\2-\1"`, for the same reason patterns are.

### Using a function as the replacement

This is the feature worth knowing:

```python
def redact(match):
    number = match.group(0)
    return number[:2] + "*" * (len(number) - 4) + number[-2:]

text = "Card 4532015112830366 used"
re.sub(r"\b\d{16}\b", redact, text)
# 'Card 45************66 used'
```

Each match is passed as a match object, and whatever you return is substituted. As a result, `re.sub` becomes arbitrary text transformation.

Another common use is normalising units:

```python
def to_upper_unit(m):
    return f"{m.group('num')} {m.group('unit').upper()}"

re.sub(r"(?P<num>\d+)\s*(?P<unit>kb|mb|gb)", to_upper_unit, "5 mb and 2gb")
# '5 MB and 2 GB'
```

`re.subn` returns the count as well, which is useful when you need to know whether anything changed:

```python
result, n = re.subn(r"\d+", "#", "a1b22c333")
# ('a#b#c#', 3)
```

## How do you keep the delimiters when splitting with re.split?

Put the delimiter pattern in a capturing group. `re.split` then returns the separators in the result list alongside the pieces:

```python
re.split(r"[,;]", "a,b;c")           # ['a', 'b', 'c']
re.split(r"([,;])", "a,b;c")         # ['a', ',', 'b', ';', 'c']
```

That is handy when you need to reconstruct the string later with the separators intact.

## What is catastrophic backtracking, and how do you avoid it?

Catastrophic backtracking is exponential slowdown when nested quantifiers give the engine too many ways to split a failing input. Avoid it by removing the nesting, or by using an atomic group or possessive quantifier on Python 3.11 and later.

```python
# dangerous
pattern = r"(a+)+$"
```

Against a run of `a` followed by `X`, the engine must try exponentially many ways to divide the `a` characters between the inner and outer quantifier before concluding failure. Measured on Python 3.14 on one Windows laptop, 18 characters took 0.18 seconds, 20 took 0.64 seconds, 22 took 2.8 seconds and 24 took 10.7 seconds. In other words, each extra pair of characters roughly quadrupled the time, so a few more characters take longer than you will wait.

The problem is nested quantifiers over overlapping character sets. The fix is to restructure rather than to micro-optimise:

```python
pattern = r"a+$"          # the same strings, no nested ambiguity
```

Since Python 3.11 you can also stop the engine backtracking into a group. An atomic group `(?>a+)+$` or a possessive quantifier `(a++)+$` fails on the same 30-character input in well under a millisecond. Both are listed in the [re module syntax reference](https://docs.python.org/3/library/re.html#regular-expression-syntax).

More realistic examples show up in patterns like `(\w+\s*)+` or `(.*,)*`. If a pattern has a quantifier inside a group that is itself quantified, look closely.

Defensive habits: prefer specific character classes to `.`, avoid nesting quantifiers, and test patterns against long non-matching input rather than only against strings that match.

## A worked example: parsing web server logs

Pulling structured records out of semi-structured text brings these advanced Python regex features together:

```python
log_line = re.compile(r"""
    ^(?P<ip>\d+\.\d+\.\d+\.\d+)\s+
    -\s+-\s+
    \[(?P<ts>[^\]]+)\]\s+
    "(?P<method>[A-Z]+)\s(?P<path>\S+)[^"]*"\s+
    (?P<status>\d{3})\s+
    (?P<size>\d+|-)
""", re.VERBOSE)

lines = [
    '203.0.113.7 - - [29/Jul/2026:14:32:05 +0530] "GET /index.html HTTP/1.1" 200 5120',
    '198.51.100.23 - - [29/Jul/2026:14:32:09 +0530] "POST /api/login HTTP/1.1" 401 -',
    "garbage line",
]

records = []
for line in lines:
    m = log_line.match(line)
    if m:
        records.append(m.groupdict())

for r in records:
    print(r)
```

Output:

```text
{'ip': '203.0.113.7', 'ts': '29/Jul/2026:14:32:05 +0530', 'method': 'GET', 'path': '/index.html', 'status': '200', 'size': '5120'}
{'ip': '198.51.100.23', 'ts': '29/Jul/2026:14:32:09 +0530', 'method': 'POST', 'path': '/api/login', 'status': '401', 'size': '-'}
```

The garbage line is skipped silently because `match` returns `None`. Note `[^\]]+` for the timestamp rather than `.+?`: a negated class cannot overshoot the closing bracket, so it is both clearer and less prone to backtracking. Every value comes back as a string, so convert `status` and `size` yourself.

## Common mistakes with advanced Python regex

- **Adding a group and breaking `findall`.** A single capturing group changes the return value from whole matches to group contents. Use `(?:...)` when you only need grouping.
- **Variable-width lookbehind.** It raises `re.error`; restructure with a capturing group.
- **Forgetting the `r` prefix on replacements.** `"\1"` is a control character, not a backreference.
- **Passing `count` or `flags` positionally to `re.sub`.** It is deprecated since Python 3.13 and easy to get wrong; use keywords.
- **Testing only strings that match.** Backtracking problems appear on long inputs that fail.



## Related reading

Regex is not always the answer: if `str.split`, `startswith` or an `in` check does the job, it is easier to read and test. [Mastering Python strings](https://www.1stepgrow.com/articles/python-strings) shows those methods. [Regular expressions in Python, Part 1](https://www.1stepgrow.com/articles/regular-expressions-in-python) covers the fundamentals. For processing large log files one line at a time, see [iterators and generators in Python](https://www.1stepgrow.com/articles/python-iterators-generators).

## Frequently asked questions

### What is the difference between a capturing and non-capturing group?

A capturing group (...) stores the matched text so you can retrieve it, and it counts toward group numbering. A non-capturing group (?:...) groups for alternation or quantification without storing anything. Use non-capturing groups when you only need the grouping, so your group numbers stay meaningful and re.findall returns whole matches.

### What is a lookahead in regex?

A lookahead is a zero-width assertion that checks whether something follows the current position without consuming it. (?=...) is a positive lookahead and (?!...) is a negative one. It is useful for matching a thing only in a certain context, such as a number followed by 'rupees', while keeping that context out of the match.

### Can lookbehind be variable length in Python?

Not in the standard re module. The documentation says the contained pattern must match strings of a fixed length, so (?<=\d{2,4}) raises re.error, as do alternatives of different widths. Restructure the pattern with a capturing group instead, or use the third-party regex module, which the re documentation points to for additional features.

### What is catastrophic backtracking?

It happens when nested quantifiers create exponentially many ways to match a string that ultimately fails, and the engine tries them all. The pattern (a+)+$ against a long run of 'a' followed by 'X' can hang for seconds or minutes. Avoid nesting quantifiers over overlapping character sets, or use an atomic group, available since Python 3.11.

---
_Source: 1stepGrow (https://www.1stepgrow.com/articles/advanced-python-regex/). Cite with the title, "1stepGrow" and a link._