Involved Source Filescompile.go Package syntax parses regular expressions into parse trees and compiles
parse trees into programs. Most clients of regular expressions will use the
facilities of package regexp (such as Compile and Match) instead of this package.
# Syntax
The regular expression syntax understood by this package when parsing with the Perl flag is as follows.
Parts of the syntax can be disabled by passing alternate flags to Parse.
Single characters:
. any character, possibly including newline (flag s=true)
[xyz] character class
[^xyz] negated character class
\d Perl character class
\D negated Perl character class
[[:alpha:]] ASCII character class
[[:^alpha:]] negated ASCII character class
\pN Unicode character class (one-letter name)
\p{Greek} Unicode character class
\PN negated Unicode character class (one-letter name)
\P{Greek} negated Unicode character class
Composites:
xy x followed by y
x|y x or y (prefer x)
Repetitions:
x* zero or more x, prefer more
x+ one or more x, prefer more
x? zero or one x, prefer one
x{n,m} n or n+1 or ... or m x, prefer more
x{n,} n or more x, prefer more
x{n} exactly n x
x*? zero or more x, prefer fewer
x+? one or more x, prefer fewer
x?? zero or one x, prefer zero
x{n,m}? n or n+1 or ... or m x, prefer fewer
x{n,}? n or more x, prefer fewer
x{n}? exactly n x
Implementation restriction: The counting forms x{n,m}, x{n,}, and x{n}
reject forms that create a minimum or maximum repetition count above 1000.
Unlimited repetitions are not subject to this restriction.
Grouping:
(re) numbered capturing group (submatch)
(?P<name>re) named & numbered capturing group (submatch)
(?:re) non-capturing group
(?flags) set flags within current group; non-capturing
(?flags:re) set flags during re; non-capturing
Flag syntax is xyz (set) or -xyz (clear) or xy-z (set xy, clear z). The flags are:
i case-insensitive (default false)
m multi-line mode: ^ and $ match begin/end line in addition to begin/end text (default false)
s let . match \n (default false)
U ungreedy: swap meaning of x* and x*?, x+ and x+?, etc (default false)
Empty strings:
^ at beginning of text or line (flag m=true)
$ at end of text (like \z not \Z) or line (flag m=true)
\A at beginning of text
\b at ASCII word boundary (\w on one side and \W, \A, or \z on the other)
\B not at ASCII word boundary
\z at end of text
Escape sequences:
\a bell (== \007)
\f form feed (== \014)
\t horizontal tab (== \011)
\n newline (== \012)
\r carriage return (== \015)
\v vertical tab character (== \013)
\* literal *, for any punctuation character *
\123 octal character code (up to three digits)
\x7F hex character code (exactly two digits)
\x{10FFFF} hex character code
\Q...\E literal text ... even if ... has punctuation
Character class elements:
x single character
A-Z character range (inclusive)
\d Perl character class
[:foo:] ASCII character class foo
\p{Foo} Unicode character class Foo
\pF Unicode character class F (one-letter name)
Named character classes as character class elements:
[\d] digits (== \d)
[^\d] not digits (== \D)
[\D] not digits (== \D)
[^\D] not not digits (== \d)
[[:name:]] named ASCII class inside character class (== [:name:])
[^[:name:]] named ASCII class inside negated character class (== [:^name:])
[\p{Name}] named Unicode property inside character class (== \p{Name})
[^\p{Name}] named Unicode property inside negated character class (== \P{Name})
Perl character classes (all ASCII-only):
\d digits (== [0-9])
\D not digits (== [^0-9])
\s whitespace (== [\t\n\f\r ])
\S not whitespace (== [^\t\n\f\r ])
\w word characters (== [0-9A-Za-z_])
\W not word characters (== [^0-9A-Za-z_])
ASCII character classes:
[[:alnum:]] alphanumeric (== [0-9A-Za-z])
[[:alpha:]] alphabetic (== [A-Za-z])
[[:ascii:]] ASCII (== [\x00-\x7F])
[[:blank:]] blank (== [\t ])
[[:cntrl:]] control (== [\x00-\x1F\x7F])
[[:digit:]] digits (== [0-9])
[[:graph:]] graphical (== [!-~] == [A-Za-z0-9!"#$%&'()*+,\-./:;<=>?@[\\\]^_`{|}~])
[[:lower:]] lower case (== [a-z])
[[:print:]] printable (== [ -~] == [ [:graph:]])
[[:punct:]] punctuation (== [!-/:-@[-`{-~])
[[:space:]] whitespace (== [\t\n\v\f\r ])
[[:upper:]] upper case (== [A-Z])
[[:word:]] word characters (== [0-9A-Za-z_])
[[:xdigit:]] hex digit (== [0-9A-Fa-f])
Unicode character classes are those in unicode.Categories and unicode.Scripts.op_string.goparse.goperl_groups.goprog.goregexp.gosimplify.go
Package-Level Type Names (total 15, in which 9 are exported)
An Inst is a single instruction in a regular expression program. // InstAlt, InstAltMatch, InstCapture, InstEmptyWidthOpInstOp // all but InstMatch, InstFailRune[]rune MatchEmptyWidth reports whether the instruction matches
an empty string between the runes before and after.
It should only be called when i.Op == InstEmptyWidth. MatchRune reports whether the instruction matches (and consumes) r.
It should only be called when i.Op == InstRune. MatchRunePos checks whether the instruction matches (and consumes) r.
If so, MatchRunePos returns the index of the matching rune pair
(or, when len(i.Rune) == 1, rune singleton).
If not, MatchRunePos returns -1.
MatchRunePos should only be called when i.Op == InstRune.(*Inst) String() string op returns i.Op but merges all the Rune special cases into InstRune
*Inst : fmt.Stringer
*Inst : context.stringer
*Inst : runtime.stringer
func (*Prog).skipNop(pc uint32) *Inst
func dumpInst(b *strings.Builder, i *Inst)
func regexp.iop(i *Inst) InstOp
A Prog is a compiled regular expression program.Inst[]Inst // number of InstCapture insts in re // index of start instruction Prefix returns a literal string that all matches for the
regexp must start with. Complete is true if the prefix
is the entire match. StartCond returns the leading empty-width conditions that must
be true in any match. It returns ^EmptyOp(0) if no matches are possible.(*Prog) String() string skipNop follows any no-op or capturing instructions.
*Prog : fmt.Stringer
*Prog : context.stringer
*Prog : runtime.stringer
func Compile(re *Regexp) (*Prog, error)
func dumpProg(b *strings.Builder, p *Prog)
func regexp.cleanupOnePass(prog *regexp.onePassProg, original *Prog)
func regexp.compileOnePass(prog *Prog) (p *regexp.onePassProg)
func regexp.maxBitStateLen(prog *Prog) int
func regexp.onePassCopy(prog *Prog) *regexp.onePassProg
func regexp.onePassPrefix(p *Prog) (prefix string, complete bool, pc uint32)
func regexp.shouldBacktrack(prog *Prog) bool
A Regexp is a node in a regular expression syntax tree. // capturing index, for OpCaptureFlagsFlags // min, max for OpRepeat // min, max for OpRepeat // capturing name, for OpCapture // operator // matched runes, for OpLiteral, OpCharClass // storage for short Rune // subexpressions, if any // storage for short Sub CapNames walks the regexp to find the names of capturing groups. Equal reports whether x and y have identical structure. MaxCap walks the regexp to find the maximum capture index. Simplify returns a regexp equivalent to re but without counted repetitions
and with various other simplifications, such as rewriting /(?:a+)+/ to /a+/.
The resulting regexp will execute correctly but its string representation
will not produce the same parse tree, because capturing parentheses
may have been duplicated or removed. For example, the simplified form
for /(x){1,2}/ is /(x)(x)?/ but both parentheses capture as $1.
The returned regexp may share structure with or be the original.(*Regexp) String() string(*Regexp) capNames(names []string)
*Regexp : fmt.Stringer
*Regexp : context.stringer
*Regexp : runtime.stringer
func Parse(s string, flags Flags) (*Regexp, error)
func (*Regexp).Simplify() *Regexp
func literalRegexp(s string, flags Flags) *Regexp
func parse(s string, flags Flags) (_ *Regexp, err error)
func simplify1(op Op, flags Flags, sub, re *Regexp) *Regexp
func Compile(re *Regexp) (*Prog, error)
func (*Regexp).Equal(y *Regexp) bool
func cleanAlt(re *Regexp)
func isCharClass(re *Regexp) bool
func matchRune(re *Regexp, r rune) bool
func mergeCharClass(dst, src *Regexp)
func repeatIsValid(re *Regexp, n int) bool
func simplify1(op Op, flags Flags, sub, re *Regexp) *Regexp
func writeRegexp(b *strings.Builder, re *Regexp)
func regexp.minInputLen(re *Regexp) int
p*Prog(*compiler) alt(f1, f2 frag) frag(*compiler) cap(arg uint32) frag(*compiler) cat(f1, f2 frag) frag(*compiler) compile(re *Regexp) frag(*compiler) empty(op EmptyOp) frag(*compiler) fail() frag(*compiler) init()(*compiler) inst(op InstOp) frag loop returns the fragment for the main loop of a plus or star.
For plus, it can be used after changing the entry to f1.i.
For star, it can be used directly when f1 can't match an empty string.
(When f1 can match an empty string, f1* must be implemented as (f1+)?
to get the priority match order correct.)(*compiler) nop() frag(*compiler) plus(f1 frag, nongreedy bool) frag(*compiler) quest(f1 frag, nongreedy bool) frag(*compiler) rune(r []rune, flags Flags) frag(*compiler) star(f1 frag, nongreedy bool) frag
A frag represents a compiled program fragment. // index of first instruction // whether fragment can match empty string // where to record end instruction
// parse mode flagsfree*Regexp // regexp height, for height limit check // number of capturing groups seen // number of regexps allocated // number of runes in char classes // product of all repetitions seen // regexp compiled size, for size limit check // stack of parsed expressions // temporary char class work spacewholeRegexpstring alternate replaces the top of the stack (above the topmost '(') with its alternation.(*parser) appendGroup(r []rune, g charGroup) []rune(*parser) calcHeight(re *Regexp, force bool) int(*parser) calcSize(re *Regexp, force bool) int64(*parser) checkHeight(re *Regexp)(*parser) checkLimits(re *Regexp)(*parser) checkSize(re *Regexp) collapse returns the result of applying op to sub.
If sub contains op nodes, they all get hoisted up
so that there is never a concat of a concat or an
alternate of an alternate. concat replaces the top of the stack (above the topmost '|' or '(') with its concatenation. factor factors common prefixes from the alternation list sub.
It returns a replacement list that reuses the same storage and
frees (passes to p.reuse) any removed *Regexps.
For example,
ABC|ABD|AEF|BCX|BCY
simplifies by literal prefix extraction to
A(B(C|D)|EF)|BC(X|Y)
which simplifies by character class introduction to
A(B[CD]|EF)|BC[XY] leadingRegexp returns the leading regexp that re begins with.
The regexp refers to storage in re or its children. leadingString returns the leading literal string that re begins with.
The string refers to storage in re or its children. literal pushes a literal regexp for the rune r on the stack. maybeConcat implements incremental concatenation
of literal runes into string nodes. The parser calls this
before each push, so only the top fragment of the stack
might need processing. Since this is called before a push,
the topmost literal is no longer subject to operators like *
(Otherwise ab* would turn into (ab)*.)
If r >= 0 and there's a node left over, maybeConcat uses it
to push r with the given flags.
maybeConcat reports whether r was pushed.(*parser) newRegexp(op Op) *Regexp op pushes a regexp with the given op onto the stack
and returns that regexp. parseClass parses a character class at the beginning of s
and pushes it onto the parse stack. parseClassChar parses a character class character at the beginning of s
and returns it. parseEscape parses an escape sequence at the beginning of s
and returns the rune. parseInt parses a decimal integer. parseNamedClass parses a leading POSIX named character class like [:alnum:]
from the beginning of s. If one is present, it appends the characters to r
and returns the new slice r and the remainder of the string. parsePerlClassEscape parses a leading Perl character class escape like \d
from the beginning of s. If one is present, it appends the characters to r
and returns the new slice r and the remainder of the string. parsePerlFlags parses a Perl flag setting or non-capturing group or both,
like (?i) or (?: or (?i:. It removes the prefix from s and updates the parse state.
The caller must have ensured that s begins with "(?". parseRepeat parses {min} (max=min) or {min,} (max=-1) or {min,max}.
If s is not of that form, it returns ok == false.
If s has the right form but the values are too big, it returns min == -1, ok == true. parseRightParen handles a ) in the input. parseUnicodeClass parses a leading Unicode character class like \p{Han}
from the beginning of s. If one is present, it appends the characters to r
and returns the new slice r and the remainder of the string. parseVerticalBar handles a | in the input. push pushes the regexp re onto the parse stack and returns the regexp. removeLeadingRegexp removes the leading regexp in re.
It returns the replacement for re.
If reuse is true, it passes the removed regexp (if no longer needed) to p.reuse. removeLeadingString removes the first n leading runes
from the beginning of re. It returns the replacement for re. repeat replaces the top stack element with itself repeated according to op, min, max.
before is the regexp suffix starting at the repetition operator.
after is the regexp suffix following after the repetition operator.
repeat returns an updated 'after' and an error, if any.(*parser) reuse(re *Regexp) If the top of the stack is an element followed by an opVerticalBar
swapVerticalBar swaps the two and returns true.
Otherwise it returns false.
A patchList is a list of instruction pointers that need to be filled in (patched).
Because the pointers haven't been filled in yet, we can reuse their storage
to hold the list. It's kind of sleazy, but works well in practice.
See https://swtch.com/~rsc/regexp/regexp1.html for inspiration.
These aren't really pointers: they're integers, so we can reinterpret them
this way without using package unsafe. A value l.head denotes
p.inst[l.head>>1].Out (l.head&1==0) or .Arg (l.head&1==1).
head == 0 denotes the empty list, okay because we start every program
with a fail instruction, so we'll never want to point at its output link.headuint32tailuint32( patchList) append(p *Prog, l2 patchList) patchList( patchList) patch(p *Prog, val uint32)
func makePatchList(n uint32) patchList
ranges implements sort.Interface on a []rune.
The choice of receiver type definition is strange
but avoids an allocation since we already have
a *[]rune.p*[]rune( ranges) Len() int( ranges) Less(i, j int) bool( ranges) Swap(i, j int)
ranges : sort.Interface
Package-Level Functions (total 36, in which 4 are exported)
Compile compiles the regexp into a program to be executed.
The regexp should have been simplified already (returned from re.Simplify).
EmptyOpContext returns the zero-width assertions
satisfied at the position between the runes r1 and r2.
Passing r1 == -1 indicates that the position is
at the beginning of the text.
Passing r2 == -1 indicates that the position is
at the end of the text.
IsWordChar reports whether r is considered a “word character”
during the evaluation of the \b and \B zero-width assertions.
These assertions are ASCII-only: the word characters are [A-Za-z0-9_].
Parse parses a regular expression string s, controlled by the specified
Flags, and returns a regular expression parse tree. The syntax is
described in the top-level comment.
appendClass returns the result of appending the class x to the class r.
It assume x is clean.
appendFoldedClass returns the result of appending the case folding of the class x to the class r.
appendFoldedRange returns the result of appending the range lo-hi
and its case folding-equivalent runes to the class r.
appendLiteral returns the result of appending the literal x to the class r.
appendNegatedClass returns the result of appending the negation of the class x to the class r.
It assumes x is clean.
appendNegatedTable returns the result of appending the negation of x to the class r.
appendRange returns the result of appending the range lo-hi to the class r.
appendTable returns the result of appending x to the class r.
can this be represented as a character class?
single-rune literal string, char class, ., and .|\n.
isValidCaptureName reports whether name
is a valid capture name: [A-Za-z0-9_]+.
PCRE limits names to 32 bytes.
Python rejects names starting with digits.
We don't enforce either of those.
repeatIsValid reports whether the repetition re is valid.
Valid means that the combination of the top-level repetition
and any inner repetitions does not exceed n copies of the
innermost thing.
This function rewalks the regexp tree and is called for every repetition,
so we have to worry about inducing quadratic behavior in the parser.
We avoid this by only calling repeatIsValid when min or max >= 2.
In that case the depth of any >= 2 nesting can only get to 9 without
triggering a parse error, so each subtree can only be rewalked 9 times.
simplify1 implements Simplify for the unary OpStar,
OpPlus, and OpQuest operators. It returns the simple regexp
equivalent to
Regexp{Op: op, Flags: flags, Sub: {sub}}
under the assumption that sub is already simple, and
without first allocating that structure. If the regexp
to be returned turns out to be equivalent to re, simplify1
returns re instead.
simplify1 is factored out of Simplify because the implementation
for other operators generates these unary expressions.
Letting them call simplify1 makes sure the expressions they
generate are simple.
maxSize is the maximum size of a compiled regexp in Insts.
It too is somewhat arbitrarily chosen, but the idea is to be large enough
to allow significant regexps while at the same time small enough that
the compiled form will not take up too much memory.
128 MB is enough for a 3.3 million Inst structures, which roughly
corresponds to a 3.3 MB regexp.
maxHeight is the maximum height of a regexp parse tree.
It is somewhat arbitrarily chosen, but the idea is to be large enough
that no one will actually hit in real use but at the same time small enough
that recursion on the Regexp tree will not hit the 1GB Go stack limit.
The maximum amount of stack for a single recursive frame is probably
closer to 1kB, so this could potentially be raised, but it seems unlikely
that people have regexps nested even this deeply.
We ran a test on Google's C++ code base and turned up only
a single use case with depth > 100; it had depth 128.
Using depth 1000 should be plenty of margin.
As an optimization, we don't even bother calculating heights
until we've allocated at least maxHeight Regexp structures.
maxRunes is the maximum number of runes allowed in a regexp tree
counting the runes in all the nodes.
Ignoring character classes p.numRunes is always less than the length of the regexp.
Character classes can make it much larger: each \pL adds 1292 runes.
128 MB is enough for 32M runes, which is over 26k \pL instances.
Note that repetitions do not make copies of the rune slices,
so \pL{1000} is only one rune slice, not 1000.
We could keep a cache of character classes we've seen,
so that all the \pL we see use the same rune list,
but that doesn't remove the problem entirely:
consider something like [\pL01234][\pL01235][\pL01236]...[\pL^&*()].
And because the Rune slice is exposed directly in the Regexp,
there is not an opportunity to change the representation to allow
partial sharing between different character classes.
So the limit is the best we can do.
maxSize is the maximum size of a compiled regexp in Insts.
It too is somewhat arbitrarily chosen, but the idea is to be large enough
to allow significant regexps while at the same time small enough that
the compiled form will not take up too much memory.
128 MB is enough for a 3.3 million Inst structures, which roughly
corresponds to a 3.3 MB regexp.
maxRunes is the maximum number of runes allowed in a regexp tree
counting the runes in all the nodes.
Ignoring character classes p.numRunes is always less than the length of the regexp.
Character classes can make it much larger: each \pL adds 1292 runes.
128 MB is enough for 32M runes, which is over 26k \pL instances.
Note that repetitions do not make copies of the rune slices,
so \pL{1000} is only one rune slice, not 1000.
We could keep a cache of character classes we've seen,
so that all the \pL we see use the same rune list,
but that doesn't remove the problem entirely:
consider something like [\pL01234][\pL01235][\pL01236]...[\pL^&*()].
And because the Rune slice is exposed directly in the Regexp,
there is not an opportunity to change the representation to allow
partial sharing between different character classes.
So the limit is the best we can do.
The pages are generated with Goldsv0.6.7. (GOOS=linux GOARCH=amd64)
Golds is a Go 101 project developed by Tapir Liu.
PR and bug reports are welcome and can be submitted to the issue list.
Please follow @Go100and1 (reachable from the left QR code) to get the latest news of Golds.