Loading

×Sorry to interrupt
Uncaught ReferenceError: addCTracking is not defined
throws at https://kb.vmware.com/resource/KM_TealiumTrackingJS?v=0.5:100:1

Refresh

Cookie Preference Center

General Information on cookies

When you visit our website, we use cookies to ensure that we give you the best experience. This information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies by clicking on the different category headings to find out more and change your settings. However, blocking some types of cookies may impact your experience on the site and the services we are able to offer. Further information can be found in our

Cookie Policy.

Strictly Necessary

Strictly Necessary

Always Active

Strictly Necessary

Strictly necessary cookies are always enabled since they are essential for our website to function. They enable core functionality such as security, network management, and website accessibility. You can set your browser to block or alert you about these cookies, but this may affect how the website functions. For more information please visit www.aboutcookies.org or www.allaboutcookies.org.

Cookie Details‎

Performance

Performance

Performance

Performance cookies are used to analyze the user experience to improve our website by collecting and reporting information on how you use it. They allow us to know which pages are the most and least popular, see how visitors move around the site, optimize our website and make it easier to navigate.

Cookie Details‎

Please confirm your settings by refreshing the page.

Functional

Functional

Functional

Functional cookies help us keep track of your past browsing choices so we can improve usability and customize your experience. These cookies enable the website to remember your preferred settings, language preferences, location and other customizable elements such as font or text size. If you do not allow these cookies, then some or all of these services may not function properly.

Cookie Details‎

Advertising

Advertising

Advertising

Advertising cookies are used to send you relevant advertising and promotional information. They may be set through our site by third parties to build a profile of your interests and show you relevant advertisements on other sites. These cookies do not directly store personal information, but their function is based on uniquely identifying your browser and internet device.

Cookie Details‎

Advertising Cookies

Consent Leg.Interest

Select All Vendors

Select All Vendors

Select All Hosts

Select All

Clear Filters

Information storage and access

Apply

Confirm My Choices Reject All Allow All

My Subscriptions

Loading

Knowledge Base

NSX Network Detection and Response - Denylist or enhance detection using custom signature intelligence (Yara and Suricata rules) (900065)

Last Updated: 6/6/2023Categories: InformationalTotal Views: 1978Language:
English

1 like this article

Subscribe to get an email
newsletter with new updates

1 people found this helpful Please provide article feedback:

Details

Introduction

NSX Lastline components can be enhanced with customer provided intelligence that will be applied in addition to the NSX Lastline provided detection. This is known as custom intelligence, and is described in the NSX Lastline documentation here (you will need to provide your account credentials to access this page).

https://user.lastline.com/papi-doc/api/html/intel/overview.html

Using custom intelligence, customers can enhance the detection behavior by adding their own network or YARA signatures, and custom IP and Domain information.

Types of custom intelligence

The NSX Lastline custom intelligence API supports the following 6 types of custom intelligence:

Denylist IPs: (sensors): An NSX Lastline Sensor will generate events when it detects traffic to these IPv4 addresses.

Denylist Domains: (sensors): An NSX Lastline Sensor will generate events when it detects DNS requests for these domains or traffic to IPs resolved from these domains.

IDS Rules: (sensors): An NSX Lastline Sensor will generate events when these IDS rules match against the monitored network traffic. In addition to the rules themselves, the custom intelligence API supports variables for the custom IDS rules as well as references for documenting the rules.

IDS Rule variables: (sensors): The custom IDS rules provided by a customer may include variables. The values for these variables are then provided as a separate feed. This allows a customer to, for example, define a rule at the global scope and specialize it on each sensor with different variable values defined at the sensor scope. These can also be set using the NSX Lastline portal user interface.

IDS Rule references: (sensors): The custom IDS rules provided by a customer may include shorthand references to resources such as the CVE archive. The custom intelligence API allows users to define the mapping between reference keywords (such as “cve”) and the URL prefix used to resolve the reference URL (such as “https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-”). These mappings are treated just like another entity type, although they can only be defined at the global scope.

Yara Rules: (engines): The NSX Lastline Engine will generate alerts when an analyzed objects matches one of the given rules. You can only add custom YARA rules to on-premise installations of NSX Lastline, not to hosted instances.

Scope of custom intelligence

The NSX Lastline Custom Intelligence API allows a user with multiple appliances to specify different intelligence for each appliance.

When adding intelligence entries through the API, users must select one of the following scopes, from the most general to the most specific:

Global Scope: The intelligence entries in the global scope apply to all of the user’s appliances (all appliances covered by all licenses)

License Scope: The intelligence entries in the license scope apply to all a user’s appliances configured with a specified NSX Lastline license (all appliances for a single license)

Sensor Scope: The intelligence entries in the sensor scope apply only to a specific sensor (single sensor)
The intelligence feed used by a NSX Lastline Sensor combines intelligence entries in the relevant global, license and sensor scopes.

Example 1:

The custom IP address denylist deployed to a Sensor is the union of all addresses in the customer’s Global IP Denylist, those in the relevant license’s IP Denylist, and those in the specific sensor’s IP Denylist.

If the same intelligence entry (e.g. a denylist IP) is present at multiple scopes that apply to the sensor, the most specific entry is selected.

Example 2:

If the IP 1.1.1.1 is in the denylist at the global scope with impact 50, and at sensor Sensor123‘s scope with impact 90, the denylist for Sensor123 will include IP 1.1.1.1 with impact 90.

Not that a single request to the add, get, del and list methods documented below always operates on a single scope.

Example 3:

If domain www.example.com is added to the Domain Denylist at the global scope, attempting to delete the domain at the license or sensor scope will have no effect: the www.example.com Domain will still be part of the Domain Denylist for all sensors, until it is removed at the global scope.

NOTE: Yara rules are applied by all Engine appliances and are thus always added on the global scope.

Please note the difference in granularity between Yara rules and the other types of custom intelligence; Yara rules apply to all the engines within a given customer license, and only apply to on-premise installations. Network signature, IP and domain intelligence can be applied globally, to all sensors in a given license, or to one or more specific sensors.

Installation

NSX Lastline uses standard RESTful APIs, so they can be used from multiple programming languages or even directly from browser plug-ins that help formulating the GET and POST requests.

To make things easier, NSX Lastline provides sample source code in the Python language, with example programs and an interactive shell that makes exploring the APIs easier.

Install python

Python is installed by default in most Linux distributions. If it is not, then please install python (version 2.7, not Python 3.x) with a command such as:

apt install python

Reading package lists... Done

Building dependency tree

Reading state information... Done

python is already the newest version.

Install NSX Lastline Python API shell and examples

Installing on NSX Lastline sensor

If installing python code directly on the NSX Lastline sensor, you will need command line access. You can either do this from the console of the sensor, or via a remote terminal session using the secure shell (ssh).

root@paris-sensor1:~# lastline_setup

Lastline Enterprise Sensor Configuration Interface

-> new_monitoring_user_password YOUR_SSH_PASSWORD

-> save

-> exit

You can now login interactively with the login monitoring and the ssh password you defined above.

Installing on a different machine

Assuming you have console or SSH access already, the only additional step is to make sure the Python path is defined (it’s already done for NSX Lastline appliances). This should be added to your .profile to persist over login sessions.

lastline@uclient:~/papi_client$ export PYTHONPATH=$PYTHONPATH:~lastline/papi_client

If you ever see the error below, it’s almost certainly because $PYTHONPATH is not defined.

lastline@uclient:~/papi_client$ python scripts/papi_shell.py
Traceback (most recent call last):
File "scripts/papi_shell.py", line 16, in <module>
from papi_client import papi_client
ImportError: No module named papi_client
Install the code from Lastline (you will need to be authenticated with your account credentials)

monitoring@paris-sensor1:~$ wget https://user.lastline.com/papi-doc/api/client/papi_client.tar.gz

…

Length: 75337 (74K) [application/octet-stream]

Saving to: 'papi_client.tar.gz'

100%[==============================>] 75,337 235KB/s in 0.3s

2017-10-27 11:00:53 (235 KB/s) - 'papi_client.tar.gz' saved

We’ve downloaded the code - now unpack the compressed archive:

monitoring@paris-sensor1:~$ tar xvfz papi_client.tar.gz

papi_client/papi_client.ini.template

…

papi_client/scripts/lastline_configure_cloud_analysis.py

monitoring@paris-sensor1:~$ ls

papi_client papi_client.tar.gz

We now have the NSX Lastline Python modules and sample source code installed and ready to use in the directory papi_client.

cd papi_client/

monitoring@paris-sensor1:~/papi_client$ ls

examples    papi_client.ini.template     README
LICENSE    papi_client_test     scripts papi_client
push_detection.ini.template streaming_client.ini.template

Authentication

To use the custom intelligence API, users must authenticate with Account-based authentication.

monitoring@paris-sensor1:~/papi_client$ cat papi_client.ini.template

[papi]

# For hosted server use the URL for the US or EMEA Data centre.

# For on-premise, use the host name or IP address of Lastline manager

url = https://user.lastline.com/papi

# Either user-based,auth_method = account

username = [email-masked]

password = ***********

verify_ssl = True

timeout = 3

Copy the template file to papi_client.ini and edit the username and password and url fields to match your portal credentials and manager if using NSX Lastline on-premise (don’t put any quotes around the username or password fields). If working with an on-premise manager, use verify_ssl = False.

monitoring@paris-sensor1:~/papi_client$ cp papi_client.ini.template papi_client.ini

monitoring@paris-sensor1:~/papi_client$ vi papi_client.ini

You have now created an authentication file for the API calls in papi_client.ini. Modifying the custom intelligence for a NSX Lastline installation (add and delete methods for each intelligence type), listing and getting feeds (list and feed methods for each intelligence type) require a user account with administrator privileges.

Viewing the configured custom intelligence (get method for each intelligence type) can be done with any user account that has access to a specified sensor or license. This permission is called “Access alerts” in the NSX Lastline Portal. When working with multiple licenses and sensors (e.g. an MSSP or complex network) you can specify which license and sensor the intelligence applies to with this parameter to the API call : 'key' : "Gxxxxxxxxxxxxxx:sensor01"

API input and output

To use an API function that is querying or listing (get or list) a value or values, the HTTP method is GET, when making a change (add or delete), the HTTP method is POST. The API documentation says which method should be used for each API function.

NSX Lastline APIs require JSON input, and will generate either JSON (default) or XML as output.

Tutorial – adding and testing a custom Suricata signature

Objective

We will make a web request before defining a custom signature and note there is no event generated.

We will then define a custom signature, check it is present, then generate the same request and confirm that the event appears in the NSX Lastline portal.

We’ll add a custom rule that detects HTTP requests with the exotic User-Agent: header ThunderChicken.

Generate web request before adding signature

To try and reach our exotic avian, we’ll craft a request from a machine (note its source IP address) with curl to a web site used for NSX Lastline testing where the web request will pass by one of the sensors:

curl -A "ThunderChicken" -O http://lastlinedemo.com/demo/testfiles

Checking network events in the NSX Lastline portal shows no event from this machine, or at least nothing that matches our chicken.

Adding a custom signature

To add a custom suricata rule, we’ll need to authenticate (1), then add the signature (2). After checking the return status, we’ll list the signatures (IDS rules) (3) to make sure it’s there.

The signature we’ll add is this:

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171026; rev:1;)

and we’ll add two other bits of information to our rule - source : custom-ids , impact : 70

1 - Authentication

You must have installed the NSX Lastline python modules and created the papi_client.ini file in the current directory with the correct credentials for this to work. Don’t forget to define the environment variable PYTHONPATH if using a non-Lastline machine (see above).

monitoring@paris-sensor1:~/papi_client$ python scripts/papi_shell.py

Loaded view 'accounting'

…

Loaded 15 views:

..

-->webrequest

---------------------------------------------------------

Lastline PAPI shell

---------------------------------------------------------

In [1]: client.intel.list_ids_rule()

Logging in...

Starting new HTTPS connection (1): user.lastline.com

"POST /papi/login HTTP/1.1" 200 33

Request took 852 ms

Completed.

Doing GET request to https://user.lastline.com/papi/intel/ids_rule/list.json

"GET /papi/intel/ids_rule/list.json HTTP/1.1" 200 26

Request took 213 ms

Out[1]: []

OK – so we’re connected, and there are no custom rules yet. Let’s go find some chickens!

2 - Add the signature

In [2]: client.intel.add_ids_rule([{'rule' : 'alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171026; rev:1;)' , 'key' : "Gxxxxxxxxxxxxxx:sensor01" , 'source' : 'custom-ids' , 'impact' : 70 }])

Doing POST request to https://user.lastline.com/papi/intel/ids_rule/add.json

"POST /papi/intel/ids_rule/add.json HTTP/1.1" 200 80

Request took 251 ms

Out[2]:{u'added': 1, u'errors': [], u'unchanged': 0, u'updated': 0}
The JSON output tells us 1 added, no errors, no unchanged and no updated.

We’ve added our signature with no errors. Last step is to check it’s there:

3- List signatures

In [3]: client.intel.list_ids_rule()

Doing GET request to https://user.lastline.com/papi/intel/ids_rule/list.json

"GET /papi/intel/ids_rule/list.json HTTP/1.1" 200 964

Request took 195 ms

Out[3]:

[{u'account': [email-masked]',

u'classification': u'trojan-activity',

u'customer_rule': u'alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171026; rev:1;)',

u'group_id': 0,

u'impact': 70,

u'last_modified': u'2017-10-27 09:07:02',

u'message': u'Wacky UA Test (ThunderChicken)',

u'refs': u'url:www.lastline.com',

u'revision_id': 1,

u'rule_id': 20171026,

u'sensor_rule': u'alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Wacky UA Test (ThunderChicken)"; flow:to_server,established; content:"User-Agent|3a| ThunderChicken"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:360000001; rev:1; gid:42; threshold: type limit, track by_src, seconds 60, count 10;)',

u'signature_id': 360000001,

u'source': u'custom-ids'}]

OK – looks good! But does it work? Let’s find out.

Generate web request after adding signature

Now we’re equipped with our chicken detector, let’s note our source and destination IP address and generate that request again:

monitoring@paris-sensor1:~$ hostname

paris-sensor1

monitoring@paris-sensor1:~$ host lastlinedemo.com

lastlinedemo.com has address 34.198.179.12

curl -A "ThunderChicken" -O http://lastlinedemo.com/demo/testfiles

Within a couple of minutes, we should see a new event in the Events section of the portal:

Yes! We’ve seen a ThunderChicken! Let’s check the details of this scary avian (Indicators of Chicken, IoC):

We can see the message, classification, source and impact match what we put in our rule.

Looking at the captured traffic, we can see our User-Agent matches our suspicious bird:

Adding multiple rules

What if we want to add multiple rules in one go? We could iterate through this process, adding one rule at a time, or if the rules share common source and impact values, we can read multiple rules from a file, like this – with a new rule, detecting deranged hounds:

monitoring@paris-sensor1:~/papi_client$ cat beagle.sig

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Zany UA Test (CrazyBeagle)"; flow:to_server,established; content:"User-Agent|3a| CrazyBeagle"; nocase; http_header; reference:url,www.lastline.com; classtype:trojan-activity; sid:20171028; rev:1;)

In [12]: client.intel.add_ids_rule([{'rule' : file('beagle.sig').read() , 'key' : "Gxxxxx:sensor01" , 'source' : 'custom-ids' , 'impact' : 80 }])
Doing POST request to https://user.lastline.com/papi/intel/ids_rule/add.json

"POST /papi/intel/ids_rule/add.json HTTP/1.1" 200 80

Request took 193 ms

Out[12]: {u'added': 1, u'errors': [], u'unchanged': 0, u'updated': 0}

Adding custom IP intelligence

Custom IP Denylist:

add_ip(): Add or update one or more IPs in the custom IP denylist

get_ip(): Get an IP’s entry from the custom IP denylist

del_ip(): Remove one or more IPs from the custom IP denylist

list_ip(): List the IPs in the custom IP denylist

feed_ip(): Get the IP denylist feed as it would be served to a sensor

NSX Lastline provides an example Python script that allows you to add lists of IP and Domain intelligence as comma separated files. The source code is provided in the folder examples.

If you create a sample text file in CSV format with 2 IP Addresses, a score, source and comment them, called sample_denylist.txt:

monitoring@sensor:~lastline/papi_client/examples$ cat sample_denylist.txt
1.2.3.4,10,Testsource,comment1
3.4.5.6,20,Testsource2,comment2

python add_custom_intel_csv.py sample_blacklist.txt "IP,IMPACT,SOURCE,COMMENT"
Parsed 2 feed entries
[{'impact': 10, 'ip': '1.2.3.4', 'comment': 'comment1', 'source': 'Testsource'},
{'impact': 20, 'ip': '3.4.5.6', 'comment': 'comment2', 'source': 'Testsource2'}
]
Adding 2 IP entries through intelligence API for key 'None'
Logging in...
Starting new HTTPS connection (1): user.lastline.com
"POST /papi/login HTTP/1.1" 200 33
Request took 1123 ms
Completed.
Doing POST request to https://user.lastline.com/papi/intel/ip/add.json
"POST /papi/intel/ip/add.json HTTP/1.1" 200 80
Request took 276 ms
{u'updated': 0, u'errors': [], u'unchanged': 0, u'added': 2}
We added 2 new entries. Now add a 3rd line to this file like this:

5.6.7.8,30,Testsource3,comment3

python add_custom_intel_csv.py sample_blacklist.txt "IP,IMPACT,SOURCE,COMMENT"
…

Parsed 3 feed entries
[{'impact': 10, 'ip': '1.2.3.4', 'comment': 'comment1', 'source': 'Testsource'},
{'impact': 20, 'ip': '3.4.5.6', 'comment': 'comment2', 'source': 'Testsource2'}
, {'impact': 30, 'ip': '5.6.7.8', 'comment': 'comment3', 'source': 'Testsource3'
}]
…

{u'updated': 0, u'errors': [], u'unchanged': 2, u'added': 1}

We added one new entry, and left 2 entries unchanged.

In [6]: Result=client.intel.list_ip()
Doing GET request to https://user.lastline.com/papi/intel/ip/list.json
"GET /papi/intel/ip/list.json HTTP/1.1" 200 503
Request took 267 ms

Check the length (it should be 3) and show 1st and 2nd elements (indices 0 and 1)

In [7]: len(Result)
Out[7]: 3

In [8]: Result[0]
Out[8]:
{u'account': [email-masked]',
u'comment': u'comment1',
u'impact': 10,
u'ip': u'1.2.3.4',
u'last_modified': u'2016-08-30 18:23:57',
u'source': u'Testsource'}

Let's query the results of a specific IP address we know is there:

In [10]: client.intel.get_ip( u'3.4.5.6' )
Doing GET request to https://user.lastline.com/papi/intel/ip/get.json
"GET /papi/intel/ip/get.json?ip=3.4.5.6&inherit_scope=False HTTP/1.1" 200 182
Request took 237 ms
Out[10]:
{u'account': [email-masked]',
u'comment': u'comment2',
u'impact': 20,
u'ip': u'3.4.5.6',
u'last_modified': u'2016-08-30 18:23:57',
u'source': u'Testsource2'}

If you want to delete something, you will need to specify it as a python array, in square brackets '[ element ]' rather than a single element.

In [12]: client.intel.delete_ip( [u'3.4.5.6','5.6.7.8','20.30.40.50'] )
Doing POST request to https://user.lastline.com/papi/intel/ip/del.json
"POST /papi/intel/ip/del.json HTTP/1.1" 200 66
Request took 265 ms

Out[12]: {u'deleted': 3, u'errors': [], u'missing': 0}

Adding custom Domain intelligence

Custom domain Denylist:

add_domain(): Add or update one or more domains in the custom domain denylist

get_domain(): Get a domain’s entry from the custom domain denylist

del_domain(): Remove one or more domains from the custom domain denylist

list_domain(): List the domains in the custom domain denylist

feed_domain(): Get the domain denylist feed as it would be served to a sensor

You can add lists of domains to the NSX Lastline domain denylist using the script described above for lists of IP Addresses, using the keyword domain rather than ip.

Adding custom IDS (Suricata) rules

At the time of writing, NSX Lastline supports the Suricata 5.0.1 version which includes support for LUA scripts. NSX Lastline will continue to upgrade the version of Suricata to follow new versions, once they have proven stable. See the tutorial above for a worked example of adding Suricata rules.

Custom IDS rules:

add_ids_rule(): Add or update one or more custom IDS rules

get_ids_rule(): Get a custom IDS rule

del_ids_rule(): Remove one or more custom IDS rules

list_ids_rule(): List the custom IDS rules

feed_ids_rule(): Get the feed of custom IDS rules as it would be served to a sensor

Adding custom IDS (Suricata) variables

Variables for custom IDS rules:

add_ids_rule_variable(): Add or update one or more variables for custom IDS rules

get_ids_rule_variable(): Get the value of a variable for custom IDS rules

del_ids_rule_variable(): Remove one or more variables for custom IDS rules

list_ids_rule_variable(): List the variables for custom IDS rules

feed_ids_rule_variable(): Get the feed of variables for custom IDS rules as it would be served to a sensor

Adding custom IDS (Suricata) rule references

Reference for custom IDS rules:

add_ids_rule_reference(): Add or update one or more references for custom IDS rules

get_ids_rule_reference(): Get the value of a reference for custom IDS rules

del_ids_rule_reference(): Remove one or more reference for custom IDS rules

list_ids_rule_reference(): List the references for custom IDS rules

Adding custom YARA rules

Note: Currently NSX Lastline appliances utilize YARA version 3.5.0

Below are the API methods for custom YARA rule:

add_yara_rule(): Add or update one or more Yara rules

get_yara_rule(): Get a Yara rule previously added to the system

del_yara_rule(): Remove one or more Yara rules

list_yara_rule(): List the Yara rules previously added to the system

feed_yara_rule(): Get the Yara rule feed as it would be served to an Engine appliance

This is an example of how a user provided YARA rule with a priority of 100 would appear:

NSX Lastline user interface for custom rule variable intelligence

The NSX Lastline portal Admin tab allows Suricata IDS Rule Variables to be viewed and defined.

As noted in the section above – Scope – the level (Global to sensor specific) can be defined for each variable and value – below is a variable specific to a single sensor “France”.

Related Information

Note: This article is applicable to the standalone NSX Network Detection and Response product (formerly Lastline) and is not intended to be applied to the NSX NDR feature of NSX-T.

Detectable by VMware SkylineTM

Actions

Copy

link to clipboard
copied!

Print

Language:
English

Attachments

Additional Resources

KB
•
Workspace ONE Boxer 21.06 for iOS での [タスク] ボタンおよびフォルダの廃止 (84470)

KB
•
Windows 10 Wi-Fi プロファイルがインストールされない (85058)

Results 1-2 of 2

Ask The Community

Get answers quickly from VMware experts in the community

Post Subject

CONTINUE IN COMMUNITIES

Related Products:

VMware NSX Detonator

VMware NSX Defender

Related Versions:

Take Our Survey

Support Assistant