I'm not even sure how to ask this question, so I'm just throwing out some details to see if anyone has any ideas about what might be happening.
I'm running RStudio Server Professional (Academic license) for use in my research group and graduate classes. For the last couple of months, I periodically login to the /admin panel, and the CPU usage will be pegged at 32 cores, and will have been sitting there for days. When I login to the server directly, and check the processes, there are always several cron jobs running from some of the users, splitting the cores evenly across them (in particular, two graduate students who are regular users of the RStudio server).
When I check with them, they aren't actually running anything, and the sessions check shows that they aren't doing anything in specific rsessions. But the only thing that these students do on this server is use RStudio. They don't login via SSH, they don't run any other services, nothing.
So does anyone have any idea what might be triggering an extremely high computational load via cron job, but not as part of an explicit rsession? It's happened half a dozen times in the last two months, and happened again this morning (and for the last three days, it turns out), and I'm out of ideas to try, aside from disabling those user accounts' access to cron jobs in an attempt to force an error or bug which reveals the culprit.
They are completely disavowing knowledge. In fact, neither even knew what cron was. But it's possible that something that they are running as code (a package, maybe?) is automatically hooking into something and running a job for them, somehow. It's just weird that it keeps coming back, despite periods where they literally aren't even logging in for a few days.
Update: I followed up with their crontab files, and all three users who have shown this behaviour had the same malware running. Somehow. The server has public key SSH only (no passwords), enforced 12-digit random alphanumeric passwords even if someone got past that ...
So not RStudio related at all, unless somehow something bootstrapped on a random package to get this installed. Which isn't anything specifically to do with RStudio at all. So ... disregard question! This one's on me!
We have noticed weird "run-away" processes on one of our servers, and the trick I use is to dig into /proc/<pid> to learn more about what the process is doing / what it's working directory is / what environment variables / etc.
It has provided some very interesting diagnostics and has created some funny conversations with teammates
This has a good overview (and Julia's Drawings are generally really interesting): /proc · julia's drawings
(Also, I posted this even though you have your solution, because in the off chance that the malware did something more sophisticated than CRON, this solution still works )
Sorry to hear about your issue, though! Sounds like someone was mining crypto currency on your dollar
Sounds like it! Thanks for the suggestion, if it ever comes back I'll try it.
Very strange that the same malware ended up on three different user's personal accounts: none of them ever login via terminal, and all only use the machine via RStudio. So I really can't see what the common vector was between the three, unless a package or download from somewhere that they all commonly used managed to do it?
Everything appeared to be localized: none of them are more than standard users, so nothing propagated out onto the rest of the machine, but all three had the same crontab and the same malware running in the same .firefoxcatche directory. Scrubbed that for all three, and manually checked the rest of the users to make sure.
I don't know how this would make any sense on a linux machine that doesn't have a desktop, but maybe a bit of insight into what was going on / what could link them...? That's super weird!
It was not! Turns out it was some kind of malware crypto-currency thing that named itself like .firefoxcache to avoid detection. Still have no idea how it got on there ...
)