 # Debugging process groups

## Outline

1. The Example: The Rank-Splitting MPI Application
2. Compiling the Rank Split application
3. Launching the applicatio with Linaro DDT
4. Group Splitting
5. Learning how to use Tracepoints
6. Learning how to use the MDA Viewer
7. Verify the correctness of the Rank-Splitting application

----

## 1. The Example: The Rank-Splitting MPI Application

The `split.c` application is designed to demonstrate how different MPI ranks can be routed to separate code paths, a common pattern in complex parallel applications.

The application uses an MPI rank comparison to split processes into two logical groups (Even and Odd) and includes an element of deliberate Load Imbalance (a sleep call) to demonstrate DDT's inspection power.

----

## 2. Compiling the Rank Split application

To compile the application, navigate to the examples directory and run the make command, which utilizes the provided Makefile.

```bash
cd examples
make
```
>The -O0 and -g flags are critical:
>-O0 disables optimization, which ensures that the debugger displays the execution flow exactly as written in the source code.

>-g includes debugging symbols, allowing DDT to map executable instructions back to lines in the simple.c source file.

----

## 3. Launching the applicatio with Linaro DDT

After compiling, you can launch the application with DDT by prepending ddt to your usual run command. In this case, we use mpiexec -n 16 to run the application with 16 processes.
```bash
ddt --connect mpiexec -n 16 split
```

Press run in the run dialog

----

## 4. Group Splitting

The goal is to stop the even and odd ranks at their respective code blocks and group them for independent control.

Set Initial Breakpoints:
Set Breakpoint 1 on Line 50 (The halt point for one set of ranks).
Set Breakpoint 2 on Line 68 (The halt point for the other set of ranks).

Click Run/Continue (▶)

**Observation**: The processes will split, with 8 processes (Ranks 0, 2, 4, ...) halted at Line 50, and 8 processes (Ranks 1, 3, 5, ...) halted at Line 68. The Stacks view reflects this by showing that 8 processes are on line 50 and 8 processes are on line 68.

![image_alt](screenshots/stackview_group.png)

**Create Custom Groups**:
Go to Locals > Right-Click the `color` variable -> Select compare across processes.

Group 1: Even Ranks (Select Ranks 0, 2, 4, 6, 8, 10, 12, 14).
Group 2: Odd Ranks (Select Ranks 1, 3, 5, 7, 9, 11, 13, 15).

Right click each group of processes -> Select Create Group

**Result**: You can now select the Even Ranks group to control only the processes at Line 50, and the Odd Ranks group to control only the processes at Line 68.

![image_alt](screenshots/group_control.png)

----

## 5. Learning how to use Tracepoints

Use Tracepoints to monitor key variables and application flow non-intrusively. This is an alternative to the print statement, where you are able to trace the value of a variable without requiring any code changes.

**Setting a Tracepoint**

Navigate to the line of code where you want execution to pause (line 73 in `split.c`).
Right-Click in the margin to the left of the line number > Select `Add Tracepoint`.

![image_alt](screenshots/add_tracepoint.png)

Select the Tracepoints window and Right-Click to Edit the Tracepoint. Log the variables `sendbuf[i]` and `i`

![image_alt](screenshots/tracepoints_variables.png)

**View the Tracepoint Log**

Click Run/Continue (▶)

The application will run, and the Tracepoint output will be populated as each rank hits the tracepoints within the Tracepoints Output window.

![image_alt](screenshots/tracepoint-output.png)

----

## 6. Learning how to use the MDA viewer

The MDA View allows you to compare the contents of buffers across different processes simultaneously to check for data mismatch.

Set a Breakpoint on `MPI-Finalize()`

Click Run/Continue (▶)

Select a Group: Select the Even Ranks group (to inspect sendbuf) or the Odd Ranks group (to inspect recvbuf).

Launch MDA View: In the Variables pane, right-click on the recvbuf variable and select View as Multi-Dimensional Array.

![image_alt](screenshots/open_mda_view.png)

Use Evaluation Expression:
To verify the data transfer is correct in one view, you can enter a comparison expression in the MDA View, such as: recvbuf[\$i] == 1804289383

![image_alt](screenshots/mda_view.png)

----

## 7. Verify the correctness of the Rank-Splitting application

> The split application is intended to perform a data transfer, Where an even rank sends data and the subsequent odd rank receives it (e.g., Rank 0 sends to Rank 1, Rank 2 sends to Rank 3, etc.). The overall correct outcome is that the data sent by the even rank must exactly match the data received by the corresponding odd rank. Use Tracepoints and the MDA view to confirm this.
