Overview
We thought it would be a good idea to put together a post to accompany our 'Why FUnLogMatic?' YouTube video. In this post, we'll dig a little deeper into why Beth is having a bad day, explore the simple exercise we presented, and take a closer look at how FUnLogMatic can be used to solve it.
Act 1 - Beth’s Having a Bad Day
Part of Beth's job results in large text files being produced.
At Fusion Unchained, we've spent a lot of time on hardware verification, and anyone who has done this can relate to the challenge of managing huge debug log files generated by simulations. Every component of the simulation environment updates its state into the output log. While there are ways to mitigate this with verbosity levels, to be on the safe side, we often prefer to have more information dumped into the log during regressions. There's nothing worse than having a regression test fail after running for over a day, only to find the log lacks enough information to debug the failure.
What makes this even more challenging is that each person working on the simulation environment needs their specific debug information included, just in case the failure is related to their code. This results in logs that are thousands of lines long, despite each engineer generally needs only a fraction of the log to do their job. Once they have extracted their pertinent information, they then have to determine whether the failure is due to their implementation, someone else's implementation, or a bug in the design being simulated.
For any hardware verification engineers reading this, you might be interested to know that Beth is examining the log output generated by the UVM Ubus example running on Aldec's Riviera-PRO simulator.
Act 2 - How Beth’s Day Improved with FUnLogMatic
There are many different ways to filter and process information, but the key is doing it efficiently while maintaining the flexibility to adapt to changes in the information that occur from one investigation to the next. Every verification engineer has their own set of commands that they use to extract and analyze failures. They create aliases for these commands and sometimes scripts to handle more advanced state-based searching. Common commands include 'grep' and 'sed,' with the output of one being 'piped' to another, each command filtering out the noise and targeting the required information.
At Fusion Unchained, we found that 80% of the time, we were repeatedly executing the same commands to understand the context of the problems we were investigating. For example, we needed to find out what configuration a simulation was run in, how the design had been programmed and what transactions had been driven/received. In most cases, that information needed 'cleaning up' to allow us to focus on the key data it conveyed.
There is a big different between this simulator generated output:
# KERNEL: UVM_INFO /home/ccavanagh/Projects/FusionUnchained/Aldec/Test/ubus/ubus/examples/ubus_example_scoreboard.sv(100) @ 357890: uvm_test_top.ubus_example_tb0.scoreboard0 [ubus_example_scoreboard] READ to empty address...Updating address : 6c22 with data : c6
# KERNEL: UVM_INFO /home/ccavanagh/Projects/FusionUnchained/Aldec/Test/ubus/ubus/examples/ubus_example_scoreboard.sv(100) @ 358180: uvm_test_top.ubus_example_tb0.scoreboard0 [ubus_example_scoreboard] READ to empty address...Updating address : dc7c with data : 80
# KERNEL: UVM_INFO /home/ccavanagh/Projects/FusionUnchained/Aldec/Test/ubus/ubus/examples/ubus_example_scoreboard.sv(100) @ 358450: uvm_test_top.ubus_example_tb0.scoreboard0 [ubus_example_scoreboard] READ to empty address...Updating address : f366 with data : c6
# KERNEL: UVM_INFO /home/ccavanagh/Projects/FusionUnchained/Aldec/Test/ubus/ubus/examples/ubus_example_scoreboard.sv(100) @ 358700: uvm_test_top.ubus_example_tb0.scoreboard0 [ubus_example_scoreboard] READ to empty address...Updating address : b0e0 with data : 49
357890 sb0 READ empty Addr: 0x6c22 Data: 0xc6
358180 sb0 READ empty Addr: 0xdc7c Data: 0x80
358450 sb0 READ empty Addr: 0xf366 Data: 0xc6
358700 sb0 READ empty Addr: 0xb0e0 Data: 0x49
Fusion Unchained's LogMatic tool supports the easy creation of these 'noise filters'. In addition it provides functionality to further process, manipulate, and highlight extracted data, thus streamlining its analysis.
A suite of searches and manipulations can be put together, saved, and reused against other files, thereby expediting the analysis process. These suites of searches can be shared with other users or further built upon as additional information is identified.
FUnLogMatic also has features to support sharing data in more readable forms, such as generating tables and reports, and highlighted text in HTML output that can be cut and pasted into emails and documents.
With the click of a button, a user can isolate useful information and easily share it with others.
Act 3 - It's Not Just for Beth
Fusion Unchained developed LogMatic to assist verification engineers, but anyone who needs to search and manipulate text or data files can benefit from its capabilities.
Engineers and scientists of all kinds constantly review data, look for patterns, spot anomalies, and transform that data into more usable forms. IT professionals regularly examine logs to assess system health and isolate issues before they impact performance and stability. Essentially, anyone who needs to process information and modify it into a more efficient form will find LogMatic invaluable.
Act 4 - Beth’s Simple Example
To keep the video concise we chose a simple yet somewhat involved exercise to set Beth.
"Find the lines beginning with red, ending in green or yellow that have blue in them somewhere, and in those lines change the red to orange and insert black in front of the blues."
red blue green yellow green
yellow red purple green blue
red purple green green yellow
red green blue red purple
purple yellow red green green
blue blue purple yellow red
red yellow yellow blue green
green purple red purple red
Although, using colors it is actually no different to a line from the ubus log file mentioned above:
# KERNEL: UVM_INFO /home/ccavanagh/Projects/FusionUnchained/Aldec/Test/ubus/ubus/examples/ubus_example_scoreboard.sv(100) @ 160: uvm_test_top.ubus_example_tb0.scoreboard0 [ubus_example_scoreboard] READ to empty address...Updating address : 68d5 with data : ab
# KERNEL: UVM_INFO /home/ccavanagh/Projects/FusionUnchained/Aldec/Test/ubus/ubus/examples/ubus_example_scoreboard.sv(89) @ 210: uvm_test_top.ubus_example_tb0.scoreboard0 [ubus_example_scoreboard] WRITE to existing address...Updating address : a212 with data : 4d
For this simple case one could put together a search that does everything in a single shot e.g.
% egrep "^red.*?blue.*?(yellow|green)$" color.log
red blue green yellow green
red yellow yellow blue green
But, that can become difficult to maintain when the search terms become more complicated, and especially when capture groups come into the equation. It becomes even more convoluted when adding in character groups to ignore inconsequential numbers between different lines of text.
At Fusion Unchained, we find that breaking the search process into manageable steps simplifies maintenance when updates occur and clarifies the overall thought process.
The simple example naturally falls into four steps. The first three steps filter out the noise and isolate the information we need. The fourth step manipulates the data into the desired form.
LogMatic is designed to support this approach of breaking up the search and applying manipulations (mapping operations) to the results. However, it can also handle more complex single-step searches. In a future post, we will explore the Regex Designer feature in LogMatic, that aids in developing more complicated searches.
Act 5 - Beth Demonstrates FUnLogMatic
The following guide walks through the steps Beth took to solve the example using Fusion Unchained's LogMatic.
Act 6 - Beth is a Happy Camper
We end with Beth being a happy camper, she no longer needs to scroll through endless lines of data. She can open a file, load a previously setup search/mapping suite and click "Run".
Beth has 'automated the boring so she can have FUn doing the hard stuff', of putting together a solution.