Parsers

This article describes the introduction of parsers and how you set up, publish, and manage parsers to normalize the raw logs and convert it into Unified Data Model (UDM) format.

Parsers normalize the raw log data that is received from different sources, and convert the data into the Unified Data Model (UDM) format. A set of default parsers is supported by Chronicle as long as the device's raw logs are received in the required format. Chronicle enables parsers to maintain versions if any changes are made.

Integrating Resolution Intelligence Cloud with Chronicle normalizes the raw log data by following the UDM data model. Data representation varies depending on the different vendors. For example, in Palo-Alto and Fortigate firewalls, the data fields are represented as 'close' and 'allow' respectively. Parser normalizes these data fields to a UDM format - "ALLOW". 

The following are the different formats that are normalized by the parsers.

• CSV: Comma Separated Values
• JSON: JavaScript Object Notation
• SYSLOG: syslog formatted message
• KV: key-value pair
• XML: Extensible Markup Language
• SYSLOG + KV: syslog header with key-value body
• SYSLOG + JSON: syslog header with key-value body
• SYSLOG + XML: syslog header with XML body
• LEEF: Log Event Extended Format
• CEF: Common Event Format

Resolution Intelligence Cloud supports these parsers to parse the raw log data. 

Adding Parsers

User Permissions Required: A Creator from these categories, such as Domain, Organization, and Tenant can configure parsers.

To configure a Parser,

1. Click the gear icon  at the top (or) hover over the hamburger icon  at the top left corner.
2. In the bottom of the left menu, click Configurations.
3. In the left menu, under Data Ingestion, click Parsers.
4. Click+ Add Parser in the top right corner.
   A New Parser editable page appears on the screen.
5. Select Log Type from the drop-down menu.
6. Check box next to Edit Timezone to select your desired timezone for the parser.
7. (Optional) Click Load Default Config in the top right corner, if you would prefer to use a default parser configuration. 
8. Log Format is filled automatically based on the Log Type that you selected.
9. Add sample file in the Log Sample File field.
10. Click Save in the top-right corner.
    It will be saved as a Draft and verified automatically. This shows the Auto sync updates toggle. By default, this is turned ON. You can view this toggle for parsers set to draft, approved and published states. 

• • Automatic Sync: Enabling auto-sync at the parent level (domain or org) ensures that all child accounts automatically receive parser updates. If auto-sync is disabled at the parent level, child accounts will not receive updates, regardless of their individual toggle settings.
  • Granular control: Child accounts can disable their auto-sync toggle to stop receiving updates, even if auto-sync is enabled at the parent level. However, to view the latest parser changes made by the parent, child accounts can use the View Updates option in the parser. This feature allows users to review and compare the parent’s changes and, if needed, manually save and publish the updated parser. See Comparing and Publishing Parser Changes Manually.

11. After saving as draft, select Send for Review in the top right corner.
The Parser goes to the Publisher for review.

Note: The parsers that are in the published state at the organization level inherit to the tenant level and display as “Available” in the UI. Once the data is available to parse, the state will be transformed to “Published”.

Publishing Parsers

User Permissions Required: A Publisher from these categories such as Domain, Organization, and Tenant can publish parsers.

To publish a Parser, 

1. Click the gear icon  at the top (or) hover over the hamburger icon  at the top left corner.
2. In the bottom of the left menu, click Configurations.
3. In the left menu, under Data Ingestion, click Parsers.
4. In the Parsers listing page, click on the Parser that is tagged as Under Review.
5. Verify the Parser and click Actions in the top-right corner of the screen.
6. From the drop-down list, click Approve if the Parser is appropriate. The Parser will be moved to Approved state once it is approved. Click Reject, if the Parser is not correct. The parser will be sent back to the Creator for changes and moved to the Draft state.
   Note: If the customer has not enabled “Approval or Reject” flow during the on boarding process, then the parser can be published directly from the draft state.
   
7. After approving a parser, Publish option is enabled in the top right corner.
8. Click Publish.
   A Parser will be moved to validating, published, available, or failed state successfully.

Note:

Google Chronicle may reject or accept based on the details provided in your Parser. If it accepts, then the parser parses the raw log data; otherwise, it does not parse.

Note: Tenants can directly know status of parsers they published. Domain and Organization users have to select their tenants from the drop-down list first, and then know the status of their respective tenant's parsers.

To know the status of your parser, 

1. From the Parsers listing UI, open your published parser.
2. In the upper right, click ellipsis icon. 
3. Click Get Status from the drop-down menu.
   A pop-up appears on the screen with ACCEPT or REJECT message.

Comparing and Publishing Parser Changes Manually

If the Auto Sync Updates toggle is enabled for the parent account, parser changes published by the parent are automatically pushed to its child accounts. However, child accounts with this toggle disabled will not receive the updates automatically. In such cases, the child accounts must manually view the updates and publish the changes. Follow these steps to compare the changes and publish the parser:

1. Click the gear icon at the top right corner, or hover over the hamburger icon at the top left corner.
2. At the bottom of the left menu, select Configurations.
3. In the left menu, under Data Ingestion, click Parsers.
4. Locate and click on the parser where changes were made by the parent account.
5. Click the View Updates option next to the Auto Sync Updates toggle.
   Note: This option is only visible if the Auto Sync Updates toggle is disabled for your account. A side panel will open, allowing you to compare the parent’s parser changes with your current version.
6. Click Compare to view the changes made by the parent.
7. Review the highlighted sections to see the differences.
8. If you agree with the changes, click Save and Publish to apply and publish the parser to Chronicle.

Testing Sample Log Data

This feature allows you to test and verify the sample logs before creating a parsing rule by providing sample logs in the given window.  Note that you can test only 10 sample logs at a time and each log sample should start in a new line.

To test the sample log data,

1. From the Parsers listing UI, open any parser that you would like to test on a sample log.
2. Click Test log sample at the top right corner of your screen.
   A side panel appears with two-halves on the screen.
3. Enter your sample log in the left side and click Run.
   The sample log result appears on the right with success or failure message.

Downloading Error logs 

The parsers UI offers you the option to download the error logs of a parser to troubleshoot the issues and resolve them quickly for smooth parsing of the log data. Note that you can download the error logs for the last 15 days only.

To download error logs,

1. From the Parsers listing UI, open any parser from where you want to download error logs.
2. Click Download Error Logs at the top right corner of your screen.
   An error log file will be downloaded in the .txt format.
3. Go to downloads in your local machine and open the log file to explore the errors and trouble shoot.

Finding Parsers

The free text search filters Parsers by text in the Parser Name and/or Log Type. There is no Search button.

Filtering Parsers

The parsers can be filtered using the following attributes, such as: 

Filter	Description
Tags	Keywords that are added to a parser. This includes tactics, techniques, malware, etc.info associated to a parser
Company	An organization that the user belongs to
Assigned To	The user who assigned to a parser
Comments	Notes added by the users
Created On	The Date and Time on which the parser is created for the first time
Created By	The user who creates the parser
Log Type	A name of a device that sends log
Log Format	Format of a log that a device sends (JSON and other formats)
Company Level	The level of an organization that the parser is filtered (Tenant - 3, Organization - 2 etc)
Parser Status	Current state of a parser (Draft/under review/reviewed/publishing/ published)
Last Updated By	The user who updated a parser recently.
Last Updated On	The time at which the last update is done to a parser
Last Updated (By Source)	Name of source on which the parser is filtered. A list of sources is shown in the filter.
Last Updated (By Platform)	Name of platform on which the parser is filtered. A list of platforms is shown in the filter.

You can add multiple filters and click Apply to filter your chosen entities. 

You may remove specific filters by clicking and then clicking the X to the right of the filter. To reset the Parsers page to its default filters, click Filter --> Clear Filters.

Sorting Parsers

The parsers feed lists all parsers (Published, Publishing and Draft) in the listing page after creating or publishing them. You can sort the parsers based on the following list of options.

Item	Description
Last Modified On	Date and Time of last change done to a parser
Log Type	A type of log file that is fed into the parser
Name	A name that is given to a parser
Most Used	Highly used parser
Created On	Date and Time when a parser is created for the first time

To rearrange parsers in descending or ascending order, click or .

For viewing left side menu, click three dots at the top right corner of screen and select Toggle Sidebar.

For sharing a URL, click three dots at the top right corner of the screen and select Share URL.