# Debugging a simple MPI application

## Related Videos
[Stepping Through Your Code with Linaro DDT](https://www.youtube.com/watch?v=I3oswXFO5GI)

----

## Outline

1. The Example: A Simple MPI Ping-Pong
2. Compiling the simple MPI application
3. Launching the application to Debug with Linaro DDT
4. Controlling the Application with Linaro DDT
    - Stepping Through Code
    - Using the Debug controls
5. Learning how to use Breakpoints
6. Learning how to use Sparklines
7. Verify the correctness of the Ping-Pong application

----

## 1. The Example: A Simple MPI Ping-Pong

The application we will be debugging, is a basic MPI ping-pong program. Each process sends a value of 42 to its neighbour (target) and simultaneously waits to receive this value from its other neighbour (source).

>**Determine Partners:**
> ```bash
> target = (rank + 1) % size;
> source = (rank - 1);
> if (source < 0)
>     source += size;
> ```
> Non-Blocking Receive value 42 from source
> ```bash
> MPI_Irecv(&recvbuf, 3, MPI_INT, source, 0, MPI_COMM_WORLD, &reqs[0]);
> ```
> Non-Blocking Send value 42 to target
> ```bash
> MPI_Isend(&sendbuf, 3, MPI_INT, target, 0, MPI_COMM_WORLD, &reqs[1]);
>```
> Wait for completion
> ```bash
> MPI_Waitall(2, reqs, MPI_STATUSES_IGNORE);

----

## 2. Compiling the simple MPI application

To compile the application, navigate to the examples directory and run the make command, which utilizes the provided Makefile.

```bash
cd examples
make
```

>The -O0 and -g flags are critical:
>-O0 disables optimization, which ensures that the debugger displays the execution flow exactly as written in the source code.

>-g includes debugging symbols, allowing DDT to map executable instructions back to lines in the simple.c source file.

----

## 3. Launching the applicatio with Linaro DDT

After compiling, you can launch the application with DDT by prepending ddt to your usual run command. In this case, we use mpiexec -n 4 to run the application with four processes.
```bash
ddt --connect mpiexec -n 4 simple
```

DDT should now have launched the Run Dialog. This dialog allows you to configure debugging options (like the number of processes, environment, and plugins). For more information on the Run Dialog options, [please see the run dialog options documentation](https://docs.linaroforge.com/latest/html/forge/ddt/get_started_ddt/running_a_program.html)

![image_alt](screenshots/launch-simple-mpi-example.png)

Click the Run button in the Run Dialog.

DDT will launch the processes and immediately halt all of them, typically just after the MPI_Init call in the program's execution, allowing you to begin debugging from a known state.

![image_alt](screenshots/ddt_stopped.png)

----

## 4. Controlling the Application with Linaro DDT

Once connected, DDT provides a set of controls to step through your program's execution.

You can control execution process-by-process or for all processes simultaneously:

![image_alt](screenshots/application_control.png)

**Play / Continue**:
Resumes execution for all processes until a breakpoint is hit or the program finishes.

**Pause**:
Stop a program from playing

**Step Into**:
Moves to the next line of source code unless there is a function call, in which case it steps to the first line of that function.

**Step Over**:
Moves to the next line of source code in the bottom stack frame.

**Step Out**:
Executes the rest of the function and then stops on the next line in the stack frame above.

**Process Grouping**:
In the top left, you can see controls to group and focus on specific processes (e.g., viewing All or focusing on a single process). [Please see focus control in the documentation](https://docs.linaroforge.com/latest/html/forge/ddt/controlling_program_execution/focus_control.html)

> Notice how the stack view changes as processes or groups of processes are stepped.

**Variables**: The midde-right section displays Local variables, allowing you to inspect the values of variables like rank, size, sendbuf, and recvbuf for the currently selected process. [Please see local variables in the documentation](https://docs.linaroforge.com/latest/html/forge/ddt/variables_and_data/local_variables.html)

![image_alt](screenshots/connected-to-mpi-example.png)

----

## 5. Learning How to Use Breakpoints

Breakpoints are essential for stopping execution at a specific line of code.

**Setting a Breakpoint**

Navigate to the line of code where you want execution to pause (line 32 in simple.c).
Click in the margin to the left of the line number. A small red dot will appear, indicating the breakpoint is set.
When you click Run (or Continue), all processes will run until they reach this line, then halt.

>It is good debugging practice to set a breakpoint on MPI_Finalize();

**Using Conditional Breakpoints**

For parallel programs, a basic breakpoint might halt all processes, which is often too much. You can make a breakpoint conditional to only stop when a specific process or condition is met.
Right-click the red breakpoint dot.
Select Edit Breakpoint.

![image_alt](screenshots/set-breakpoint.png)

In the condition field, you can enter a C/C++ expression, such as `source < 0` on line 23 of `simple.c`.

>Notice the stack view shows that 2 processes are stopped on MPI_Finalize(), 1 process is on MPI_Waitall() and 1 has stopped on the conditional breakpoint. This is because the process on MPI_Waitall() is waiting for its neighbour to send it the value of 42 before it is able to continue. This Neighbour has stopped on the condition and therefore has not sent this value yet.

![image_alt](screenshots/stackview.png)

----

## 6. Learning How to Use Sparklines

Sparklines are a visualization tool in DDT for tracking variable values across multiple processes simultaneously. They are extremely useful for identifying data imbalances or unexpected variations in parallel code.

**Using Sparklines**

Right-click on a variable you want to monitor across all processes (e.g rank, source, target).

Then add to evaluations by dragging and dropping the variable into the evaluations window.

A small graph (a "Sparkline") will appear next to each variable in the Evaluate window.

>The x-axis of the sparkline represents the process ID (rank).\
>The y-axis represents the variable's value for that process.

This allows you to see at a glance if all processes have the expected or equal values for a given variable. For instance, in our `simple.c` example, you could watch recvbuf to ensure all processes successfully receive the correct value after MPI_Waitall().

![image_alt](screenshots/sparklines.png)

----

## 7. Verify the correctness of the Ping-Pong application
        Run to line 32
        Drag and drop source, rank, target to Evaluations Window
        Check sparklines to verify correct MPI rank setup
        Run to line 43
        Check that sendbuf in the locals tab contains the value 42

        ddt --offline -o offline-debugging.html --break-at=simple.c:32 --break-at=simple.c:43 mpiexec -n 4 simple
