Automated parsing of profiles/traces

dabele · March 31, 2025, 5:53pm

I have a couple of trace files produced by precice (Performance Analysis. I would like to extract data from them in an automated way.

I have found the python API of perfetto, which allows SQL or Pandas Queries. Overall this is pretty flexible.

Now I’m wondering if there is a way to uniquely identify the end or beginning of a coupling window, some event that happens exactly once (or maybe a fixed amount of time) per window? The idea is to do analysis of e.g. average coupling window runtime but ignore the first N windows because they are unrepresentative of behavior in longer runs. Or similar things.

If there is no such thing, maybe another level of events could be added between the _GLOBAL level and advance level that represents coupling windows?

DavidSCN · April 1, 2025, 7:27am

I found the setup we use in aste pretty helpful, see for example this code snippet we use for the automatic analysis

github.com/precice/aste

tools/mapping-tester/gatherstats.py

dfb15a791


      
          try:
              subprocess.run(
                  ["precice-profiling", "merge", "--output", json_file, event_dir],
                  check=True,
                  capture_output=True,
              )
              subprocess.run(
                  ["precice-profiling", "export", "--output", timings_file, json_file],
                  check=True,
                  capture_output=True,
              )
              import polars as pl
          
              df = (
                  pl.read_csv(timings_file)
                  .filter(pl.col("participant") == "B")
                  .select("event", "duration")
              )
              return {
                  "globalTime": df.filter(pl.col("event") == "_GLOBAL")

This file has been truncated. show original

You can easily filter more or compute means, for quantities, e.g., the following computes the mean time and extracts the maximum across all ranks

github.com/davidscn/aste

tools/mapping-tester/gatherstats.py

cd6a09938


      
          "MeanMapDataAtTime": df.filter(
              pl.col("event").str.contains(
                  r"mapAndReadData\.A-Mesh.*\.mapConsistentAt\.FromA-Mesh$"
              )
          )
          .group_by("rank")
          .agg(pl.col("duration").mean().alias("duration_mean"))
          .select(pl.col("duration_mean").max())
          .item(),

Now I’m wondering if there is a way to uniquely identify the end or beginning of a coupling window, some event that happens exactly once (or maybe a fixed amount of time) per window?

We have an advance event (as you already mentioned) as well as a solver.advance event which tracks the time your solver utilizes. If you know your subcycling configuration, I could imagine that these are sufficient to analyze what you want.
We also have separate events for most API functions, but it could be that you have to upgrade to v3.2.0 (to be released probably this week) to get all those.

fsimonis · April 1, 2025, 8:19am

Hi,

I wasn’t aware of the perfetto python API. Thanks for sharing the knowledge!
What I normally do it to use precice-profiling export and then load the csv into polars or pandas.
This allows for straight-forward filtering and grouping based on the participant and rank.

There is currently no easy way of extracting this information from the profiling data.
The problem is that this is highly case dependent.

preCICE supports mixing explicit and implicit coupling schemes as well as mixing different time-window sizes in these coupling schemes. This makes an implementation of this pretty tricky. Maybe one could attach time-window and iteration counters to each coupling scheme advance? But there is currently no event for this.

@IshaanDesai Is currently trying to figure out the exact same issue to simplify data processing of user-defined profiling data in the micro manager.

Best
Frédéric

Topic		Replies	Views
How Is the Exchanged Data of Subcycles Generated? \| Non-Matching Timesteps & Fixed Time Window Is preCICE for me? time-stepping	3	338	August 31, 2021
Exchange data in each time step Using preCICE adapters , time-interpolation , time-stepping	5	118	April 19, 2024
preCICE time step setting for parallel-explicit coupling scheme Is preCICE for me? time-stepping , coupling-schemes	1	29	March 19, 2025
OpenFOAM adapter output during FSI simulation Using preCICE openfoam , fsi	3	335	August 14, 2021
Participant reads zeros for mapped data Using preCICE data-mapping	12	582	March 15, 2022

Automated parsing of profiles/traces

Related topics