A Quick Introduction to Regular Expressions for Security Professionals

Photo by Brett Jordan on Unsplash

Regex skills are probably one of the most underrated security skills.

Working in security often means that you have to sift through large amounts of information in the form of log files or Internet packets. You might also need to design firewall rules, set up malware scanners, or analyze traffic coming from the Internet.

Learning and mastering regex thus becomes one of the most powerful skills that you can possess as a security professional. Today, let's go through some of the most useful regex tips for security people and how you can use them to automate your most complex tasks!

Regex Syntax Overview

A regular expression, or “regex”, is a special string that describes a search pattern.

You can think of regex as consisting of two different parts: constants and operators. Constants are sets of strings, while operators are symbols that denote operations over these strings. These two elements together make regex a powerful tool of pattern matching.

For example, the regular expression below matches every IP address from subnet 192.168.0.0/24. The highlighted portions are constants, meaning that the regex will match the highlighted strings literally. The rest of the regex are operators: they have special meanings and add flexibility to the pattern matching.

192\.168\.0\.\d{1,3}

Different Regex Flavors

Before we dive into the basics of regex syntax, please note that regex has many different versions. Different software and regex engines will often have their own specificities, and it's best to check the official documentation pages for a full reference of the regex version that you are using. However, all regex tends to build upon the same set of generic rules.

Regex Operators

First off, these regex operators match with single characters:

• \d: matches any digit

• \w: matches any character

• \s: matches any whitespace

• .: matches with any single character

• \: escapes a special character

We also have a number of operators that specify the number of characters we are matching:

• *: zero or more

• +: one or more

• {3}: exactly three times

• {1, 3}: one to three times

• {1, }: one or more times

• [abc]: one of the characters within the brackets

• [a-z]: one of the characters within the range of a - z

• (a|b|c): either a or b or c

There are a lot more advanced regex features that you can use to perform more sophisticated matching. However, the simple set of operators above serves well for most security purposes.

For a complete guide to regex syntax, read RexEgg's cheat sheet.

Harness the Power of Regex

So what can we do with regex? Here are just a few of the many use cases of regex in your day-to-day tasks!

Logfile Analysis with Regex

One of the ways you can use regex is to perform complex text searches. This means regex is very useful during the analysis of log files: instead of searching for simple terms, you can use regex to quickly find more accurate results.

For example, let's say that your logfile entries are in this format:

Mar 2 12:55:09 HOST_MACHINE PROCESS LOG_MESSEGE

With regex, we can quickly find all the processes that ran during a specific time frame. This regex will match with all log entries that have the timestamp between 12 and 2 PM on March 2nd.

Mar 2 1[2-4]:[0-9]{2}:[0-9]{2}

You can also use regex to find all the IP addresses that show up in access logs. You can do something like this, which will match with all IP addresses in the log file.

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Combine a couple of different metrics (IP ranges, timestamp, hostnames, and usernames) and you'll have an extremely powerful log analysis utility that you can fully customize!

Firewall Rules and Input Validation with Regex

Many people use regex to specify firewall rules. For example, you can use regex to create rules to block requests to certain file types. This regex will match with any request that contains the terms "json", "exe", "tar" and "rar".

(json|exe|tar|rar)

A sound firewall rule will use a regex pattern like the above but with a wide range of file types, while also accounting for possible bypasses such as case changes and the inclusion of non-ASCII characters.

If you are a developer, you will also often need regex to deal with input validation in your programs. For example, the code below will reject any user input that contains non-alphanumeric characters and is longer than 50 characters.

check_username:
 if regex_match("[a-zA-Z0-9]{1,50}", user_input):
   // Continue processing if the input is valid
 else:
   // Stop processing and reject the request

Proxy Rules with Regex

Regex can also be useful when you debug or test your applications. For example, using effective regex to filter traffic on debugging proxies can make your work a lot more efficient. Instead of churning through endless requests flowing through your proxy windows (which is a gigantic time-suck), you can isolate the requests going to a specific subdomain of your site like this:

[A-Za-z0–9]*\.dev\.site\.com

Malware Scanning with Regex

Finally, regex is also one of the most powerful tools used for identifying malware.

For example, YARA is a tool that identifies malware by creating descriptions that look for certain characteristics. It uses regex patterns to detect specific text or binary patterns in files that might indicate that the file is malicious.

rule example_malware : example
{
  meta:
    description = "This is just an example."
    threat_level = 3
    in_the_wild = true
  strings:
    $a = /evil software version: [0-9a-zA-Z]{32}/
    $b = "Malware Inc"
  condition:
    $a and $b
}

This example rule states that any file that contains the strings "Malware Inc" and "evil software version: [0–9a-zA-Z]{32}" is suspected to be a piece of malware.

To learn more about how YARA detects malware, read my Intro to Malware Detection Using YARA.

Useful Regex Resources

Here are a few resources to help you build your regex skills!

To build solid regex skills, follow these amazing regex tutorials.

For some practice writing regular expressions, play the RegexOne game.

Sometimes, you can't be sure if your regular expression matches exactly what you are looking for. So to test your regex strings, use the Regex101 regex tester. And here's a great regex cheat sheet if you ever forget what a particular operator means.

Finally, don't forget to check out the documentation of your particular regex dialect before you dive into constructing regex strings!