Plotting data with a delay to simulate live stream

shiny
#1

Hello,

I am receiving new data every 3 seconds, the data contains enough points to plot a new point 10x per second. I am wanting to create a plot that appears to be live stream.

My current code is this:

  output$plot1 <- renderPlot({
   autoInvalidate()
    
 	chuck1 <- read.csv("logfile1.txt",header=FALSE) #read data coming into logfile1.txt
    autoInvalidate()
 	chuck1 <- na.omit(chuck1)
 	row_value1 <<- tail(chuck1$V1, 30) # Get most recent entries in the logfile1.txt for column 1
 	row_value2 <<- tail(chuck1$V2, 30) # Get most recent entries in the logfile1.txt for column 2
 	for(i in 1:30) #Create a for loop to cycle through the 30 entries and plot each new entry
 	{
 	  row_value3 <<- row_value1[i] # set a value equal to each iteration of the for loop to be plotted later.
 	  row_value4 <<- row_value2[i] # set a value equal to each iteration of the for loop to be plotted later.
    	plot(row_value3,row_value3, type="o", xlab="Time", ylab="Movement", main="Without", ylim=c(0.5,4), xlim = c(row_value1[1],row_value1[30]))
    Sys.sleep(0.1) # Sleep for 0.1 second
    i <- i + 1 # cycle through the loop (Unsure if i needed this just added it in just incase and tried it without it)
  }
  })

Currently this code results in nothing being plotted, I can see that even if it did work it would likely re-plot each time and the final thing I see is just 1 point on the screen but I don't even seem to be getting that.

I have found a few example online but none that quite match what I am looking for.
Any help would be greatly appreciated.

Thankyou

#2

can you post a few example lines from the text file? If the data's proprietary, put in dummy values, but it would be helpful for us to be able to reproduce what you're experiencing.

#3

The contents of the file is: Time, accellerometer data.

3449.988,1.009042
3450.137,1.008458
3450.286,1.012932
3450.434,1.009818
3450.582,1.014727
3450.731,1.011774
3450.879,1.004503
3451.028,1.004276
3451.176,1.005803
3451.324,1.005126
3451.473,1.010304
3451.621,1.001998
3451.77,1.005491
3451.918,1.001365
3452.067,1.015092
3452.215,1.003702
3452.364,1.006055
3452.512,1.009469
3452.661,1.000678
3452.809,1.01441
3452.957,1.007657
3453.105,1.01

There is a Python script that keeps the file within 100 rows for the incoming data, it writes over the last line. This keeps the last row number always 100.

#4

I have made another attempt with comments in the code that describe what I am trying to do and my line of thinking. For some reason data_main only has 91 obs. at the end of the for loop, my expectation was for each iteration of i to be added to data_main, therefore returning it to 100 while plotting it. It plots all values until the last 10 values, it skips all those in the loop except the very last value.

  output$plot5 <- renderPlot({
    autoInvalidate()
    data1 <- read.csv("logfile.txt",header=FALSE)                                 #Read data
    autoInvalidate()
    data1 <<- na.omit(data1)                                                      #remove n/a's
    data_tail <<- tail(data1$V1, 10)                                              #prepare the recent 10 entries of V1
    data_tail2 <<- tail(data1$V2, 10)                                             #prepare the recent 10 entries of V2
    i <<- 1                                                                       #declare starting value for i loop
    data3 <<- head(data1,-10)                                                     #create new dataframe with that is trimmed of the new entries
    for(i in 1:10)                                                                #Create a for loop to add those removed entries back in 1 by 1
    {
      
      data2 <<- data.frame(V1 = data_tail[i], V2 = data_tail2[i])                 # data2 gets set to each iteration of i
      data_main <<- rbind(data3, data2)                                           #each iteration of i gets appended to the end of data3 (adding entries back in)
      plot(data_main$V1,data_main$V2, type="o", xlab="Time", ylab="Movement", main="With", ylim=c(0.5,4), xlim =c(11992,11995)) #Plot (to current xlim for non-live data test)
      i <- i + 1                                                                  #Cycle through i
    }
  })

The data set is the same as above, except I made sure to use one that had exactly 100 rows for this trial

#5

Solved part of it, but it just plots everything at once still, is we use the sys.sleep, it just waits the full cycle (if it was 1 second wait for 10 iterations it waits 10 seconds and plots everything). So that makes me believe it needs a different approach or something? I feel like what I am trying to do is a reasonably simple task.

output$plot5 <- renderPlot({
    

    autoInvalidate()
    data1 <- read.csv("logfile.txt",header=FALSE)                                 #Read data
    autoInvalidate()
    data1 <<- na.omit(data1)                                                      #remove n/a's
    data_tail <<- tail(data1$V1, 10)                                              #prepare the recent 10 entries of V1
    data_tail2 <<- tail(data1$V2, 10)                                             #prepare the recent 10 entries of V2
    #i <<- 1                                                                       #declare starting value for i loop
    data3 <<- head(data1,-10)                                                     #create new dataframe with that is trimmed of the new entries
    for(i in 1:10)                                                                #Create a for loop to add those removed entries back in 1 by 1
    {
      
      data2 <- data.frame(V1 = data_tail[i], V2 = data_tail2[i])                 # data2 gets set to each iteration of i
      if (i==1){data_main <- rbind(data3, data2)}else{data_main <- rbind(data_main,data2)}
      data4 <<- data_main
      #each iteration of i gets appended to the end of data3 (adding entries back in)
      plot(data_main$V1,data_main$V2, type="o", xlab="Time", ylab="Movement", main="With", ylim=c(0.5,4), xlim =c(11992,11995)) #Plot (to current xlim for non-live data test)
                                                                       #Cycle through i
      
    }
  })
#6

you mention Sys.sleep in your narrative, but it's not in your code. If you add Sys.sleep(1) to the end of your loop, what's missing? It seems like that results in what you were after.

#7

I've tried that, I've tried putting Sys.sleep(1) in all kinds of places in that code. When I put Sys.sleep(1) anywhere in that loop it will take 10 seconds before the plot appear, note this plot has all points rather than 1 at a time. No idea why it seems to complete the entire 1:10 of the loop before displaying anything but that is whats happening.
Basically,
What I expect:
Collect data from file
Remove last 10 rows from dataframe to be plotted
Add the 10 rows back into the data frame 1 by 1 and plot it
Wait 1 second, repeat

What happens:
Collect data from file
Remove last 10 rows from dataframe to be plotted
Wait 10 seconds and plot entire dataframe (with the 10 rows already added back in)

#8

Here's a couple of thoughts... you're junk is all wrapped up in a function which is fine later, but sucks for debugging. Pull it out so you can interact easier and test things. You have this renderPlot thing which I have no idea what it is. same with autoInvalidate(). I killed that stuff so I could run your code, then added in simulated data. I put a Sys.sleep(1) at the end of the loop and it seems to do exactly what you say you want:


data1 <- data.frame(V1=1:20, V2=rnorm(20))                            #Read data

data1 <<- na.omit(data1)                                                      #remove n/a's
data_tail <<- tail(data1$V1, 10)                                              #prepare the recent 10 entries of V1
data_tail2 <<- tail(data1$V2, 10)                                             #prepare the recent 10 entries of V2
                                                                     #declare starting value for i loop
data3 <<- head(data1,-10)                                                     #create new dataframe with that is trimmed of the new entries
for(i in 1:10)                                                                #Create a for loop to add those removed entries back in 1 by 1
{
  data2 <- data.frame(V1 = data_tail[i], V2 = data_tail2[i])                 # data2 gets set to each iteration of i
  if (i==1){
    data_main <- rbind(data3, data2)
  }else{
      data_main <- rbind(data_main,data2)
  }
  data4 <<- data_main
  
  plot(data_main$V1,data_main$V2, type="o", xlab="Time", ylab="Movement", main="With") #Plot (to current xlim for non-live data test)

  Sys.sleep(1)
}

so what's missing?

#9

Thats given me a few ideas and made me realise the need to explain more of the situation, cheers for sticking around and helping.

The reason for the renderplot was because we are having several plots in rows down a page. We have been using fluidrow() to allow for this, but I am starting to feel like we may need to take a different approach.

This is the start of my code:

library(shinydashboard)
library(readr)
library(ggplot2)

Shinyserver <- function(input, output, session) {
  # Output sidebarpanel detail to UI  -------------------------------------------

  
  
  
  # Output body detail to UI  -------------------------------------------    
  output$body <- renderUI({
   
    div(width=12,
        
        # Info Boxes --------------------------------------------------------------
        
       
        fluidRow(tags$style(type="text/css", ".recalculating {opacity: 1.0;}"),
	  box(height= 400, width=10, valueBoxOutput("plot1")),
          valueBoxOutput("activity1", width = 2)),
	  fluidRow(tags$style(type="text/css", ".recalculating {opacity: 1.0;}"),
	           box(title=h3("ActivityA02"), height= 400, width=10, plotOutput("plot2", height=250)),
	           valueBoxOutput("activity2", width = 2)))
    
     })

I have some live streaming code where the external data file gets updated row by row which makes my current approach actually work. Unfortunately this new data isn't coming in 1 at a time :(.

Essentially I need to be able to place to plot in a box so that I can arrange my display to fit in with other things that are happening.

I hope I have explained myself a bit better, really appreciate the help.

Cheers.

#10

Ohhh.. I see now that this is tagged shiny and I had not paid attention to that initially! My mistake. That really changes things. I have very limited Shiny experience so I'll make a few comments and then maybe someone will come in and correct me if I'm wrong :slight_smile:

Shiny expects your R code to produce a static graph and then it renders the graph. So what you're seeing is that Shiny waits until R is totally done running everything then it renders the result. Thus the wait for 10 seconds before plotting.

Try looking at this example to see if it helps: https://stackoverflow.com/a/35346846/37751

or possibly use reactivetimer to get the same effect as the Sys.sleep: https://stackoverflow.com/a/43344162/37751

The first link above may be only for animations where the data does not change. I think the reactivetimer example is more what you need. But I am out of my experience base here...