Plotting solver output in real time

[msiple]

In my shiny app, I have a function that uses a solver to estimate a value. While the user is waiting, I want them to see the guesses from the solver plotted as the solver tries them. I can do this in R by just putting points() inside the function being solved.

When I put the function in a plotOutput object in shiny, it shows the plot all at once in the app. Can anyone advise on the best way to get the points() to show up as they're being generated? I'm open to anything!

Below is a toy example:

#> Toy example
library(shiny)

liveplot <- function(L = 5){
  plot(1:L,1:L,type='n')
  for(i in 1:L){
    p <- rnorm(1, 5, 1)
    Sys.sleep(time = 0.5)
    points(i,p)
  }
}
# to see the "animated" plotting I'm looking for, just run `liveplot()` alone.

library(shiny)

# UI
ui <- fluidPage(
   
   titlePanel("Incremental plotting"),

   sidebarLayout(
      sidebarPanel(
         sliderInput("length_i",
                     "Steps to plot:",
                     min = 1,
                     max = 10,
                     value = 6)
      ),
      
      # Plot where I would like to show points being plotted as the function generates them:
      mainPanel(
         plotOutput("livePlot")
      )
   )
)

# Server
server <- function(input, output) {
   output$livePlot <- renderPlot({
     liveplot(L = input$length_i)
   })
}

shinyApp(ui = ui, server = server)

[dano]

Huh tricky! I think the problem is that renderPlot waits until the function it is being used on finishes before actually rendering the plot, at which point your plot is done. How long does the solver take to run? If you're just wanting to visualize the process, you could write the incremental steps of the solver to a log file (e.g. solver.csv), then once it's done read the log file in and use gganimate to make an animation of the steps. That's assuming you're using something like nlminb. If you're writing the solver manually you wouldn't need to write to a log. But obviously that doesn't help you if the solver takes a long time and you're trying to use this to give the user something to do while they wait.

[msiple]

Thanks! Yeah, it is super tricky!

I do want people to see the solver output while it's working (it's a uniroot solution and it takes a looong time). But I like your suggestion for using gganimate in the meantime. It seems like it's a shiny thing-- hard to get around the fact that it waits for the process to be done before rendering. And that you can't just plot over it multiple times because the number of renderPlot() objects changes depending on how many solutions uniroot tries.

Do let me know if you think of anything else!

[barret]

Ok. This was a fun one.

Major assumption: You can iteratively call your method. Such as run_iteration()

I let the print statements in the code so that you can see what is working when. Remove them when you feel comfortable with the code.

The main trick I used was to update your data object (dt) at each step and call invalidate later to trigger the next calculation step. Invalidate later will execute on the next "tick" within shiny. This allows the plot to be printed before the next iteration is calculated.

isolate() calls were used in the calculating observe to avoid a recursive trigger of the observer. Only a change to calculate should initially trigger the observer. The subsequent triggers are caused by the invalidateLater call.

library(shiny)
library(ggplot2)

# "optimization" code
run_iteration_i <- function(n, i = 1, current_data) {
  cat("running iteration: ", i, "/", n, "\n")

  # make new data
  new_dt <- rbind(
    current_data,
    data.frame(x = i, y = rnorm(1, 5, 1))
  )

  # return new data
  new_dt
}


# UI
ui <- fluidPage(

   titlePanel("Incremental plotting"),

   sidebarLayout(
      sidebarPanel(
         sliderInput("length_i",
                     "Steps to plot:",
                     min = 20,
                     max = 500,
                     value = 30),
         textOutput("status")
      ),

      # Plot where I would like to show points being plotted as the function generates them:
      mainPanel(
         plotOutput("livePlot")
      )
   )
)

# Server
server <- function(input, output) {
  
  # data to print
  dt <- reactiveVal(NULL)
  observe({str(dt())})

  # plot to print
  p <- reactive({
    # require data to not be NULL
    req(dt())
    ggplot(dt(), aes(x, y)) + geom_point() + xlim(1, input$length_i)
  })

  # print plot
  output$livePlot <- renderPlot({
    cat("making plot\n")
    p()
  })

  # bool of "is calculating"
  calculate <- reactiveVal(FALSE)

  # iteration position
  pos <- reactiveVal()
  observe({cat("pos: ", pos(), "\n")})

  # when the input changes, reset everything and start calculating
  observeEvent({input$length_i}, {
    print("resetting")
    pos(1)
    dt(NULL)
    calculate(TRUE)
  })

  # Only called when calculate() changes or invalidateLater is triggered
  observe({
    if (!calculate()) {
      req(FALSE)
    }

    # do not track any reactive changes that occur in here
    isolate({
      # run next step
      new_data <- run_iteration_i(input$length_i, pos(), dt())
      dt(new_data)

      # increment position
      pos(pos() + 1)
    })
    
    if (pos() > input$length_i) {
      # stop calculating
      calculate(FALSE)
    } else {
      # call the method again in 0.1 seconds... gives time for plot to appear
      invalidateLater(0.2 * 1000)
    }
  })

  output$status <- renderText({
    if (calculate()) {
      paste0("Working on: ", pos(), "/", input$length_i)
    } else {
      "Ready to calculate!"
    }
  })

}

shinyApp(ui = ui, server = server)

[msiple]

Wow, this is awesome... thank you! I am still trying to figure out if it will work with the function I have, which uses uniroot() to solve for a value. It has a pointer embedded in it that saves the guesses that uniroot() makes on its way to the solution. In the meantime, I think this code will be used by many people in the future-- animating processes in real time is a cool functionality, especially for teaching. I will post again when I've tried it with the solver.

[msiple]

Note: even though my particular case is non-iterative, I'm going to tag this as a solution because I'm sure other people with iterative plotting desires will find that it perfectly resolves their issue.

This is totally awesome, thanks so much!