Which tools do you use for analytics projects? [Thread]

workflow

#1

Let’s geek it out for a bit.

Below, post your set up for working on analytics projects where you use R.

Here is mine.

OS

  • macOS on MacBook Pro

Text editor

  • Emacs with ESS for R
  • I am big on keyboard shortcuts, that is why I spend a lot of time in Emacs
  • I use RStudio occasionally, to see what features it has. I do like the Data viewer, which shows variable labels for data from SAS
  • In Emacs, I can open remote server files from local session
  • Git integration: emacs keyboard shortcuts for git
  • Color theme: solarized

Command line

  • Terminal, Red Sands theme (only looks nice on retina screen)
  • Bash
  • Tmux to handle different windows

Version control

  • Git
  • Getting more familiar with git on Emacs, but still use command line git

Files

  • Starting to use Forkflift instead of Finder, which has dark mode and split screen, but most importantly, awesome remote access

Messaging

  • At work, Slack and Hangouts

Analysis notes

  • R for most everything. If I need to do ML, I process data with R then go to python sklearn. I don’t work with big data.
  • I mostly write R scripts for processing/running analyses, and use Rmd for sharing reports
  • Use Makefile to specify project recipe

Misc

  • Alfred instead of spotlight. I live on the keyboard, and Alfred offers fast switching between apps (cmd-space, f for Finder, cmd-space, e for Emacs, etc), good file finder, and sorting for “recent” apps/files

Links

Obligatory screen shot attached


#2

Not trying to ruin this thread, just representing the simple life. :grinning: I still do important stuff like this.

OS
Windows 10 on Dell (but I told the IT department to make it a super good one)

Text editor
RStudio

Command line
No thank you

Version control
GitHub
SourceTree

Messaging
Email?

Analysis notes
R Notebooks

Everything else
RStudio


#3

Here is my current set up:

OS

  • macOS High Sierra - MBP 2015 Retina

Text Editor

  • Rstudio - I use it for mostly everything these days. Especially after tuning in my LaTeX homework and report templates there is rarely a reason I need to leave Rstudio.

  • Sublime Text w/ Emacs Key Bindings - For when I need to do quick edits to files or work with another language like Python or Haskell.

  • Emacs - Still trying to get onboard with emacs, definitely like the key bindings and shortcuts but haven’t been able to sit down and fully customize it yet to my liking.

Command Line

  • ZSH - [oh-my-zsh]
  • iTerm - set up to a global key binding that drops down from top of screen covering 1/8 of screen. Good for quick terminal use and moving around files.

Version Control

  • Git from command line

Messaging

  • iMessage
  • Slack

Misc

  • Itsycal - great light weight calendar app, makes it easier to look ahead in the month.
  • Spark for my email client - works pretty well.
  • Spotify for music

dotfiles


#4

OS:

  • Work: Windows
  • Home: Ubuntu

Text Editor:

  • Rstudio - love Rstudio so its my sole tool for creating rscripts usually

  • Sublime Text - python scripts, sql scripts, basically anything not R related I use if I need outside tools for project
    Command Line:

  • Work: Command Prompt - Painfully learned how to use the Windows command prompt because I was so use using it at home

  • Home: Terminal

Version Control:

  • Mostly GitHub
  • Sometimes BitBucket if the repo is private

Messaging:

  • Slack

Misc

  • Database: PostgreSQL, Jupyter Notebooks

#5

OS

  • Fedora 26 (99% of the time). Dotfiles synced to S3 bucket
  • Weeeenders 10 when I have to use MS Office. I’m on a dual-boot setup running on a Franken-Dell laptop (about every part has been replaced at some time)
  • Cloud, I really do a lot of stuff on remote servers. Digital Ocean is really nice.

Text Editor /IDE

  • RStudio: 95% of my R work is in RStudio. I administer and do my analysis work on a cloud RStudio Server rather than a local instance. It makes data IO much easier.
  • Pycharm: The Python equivalent of what RStudio is to R and I use it a LOT
  • Sublime Text for general use
  • Visual Studio Code has been very useful as a good general purpose IDE. I’ve used it to write, test, and compile/run short snippets in everything from Scala to Golang and for just fooling around, it’s great.
  • Emacs. I love the idea of Emacs and I’ve seen how productive people can be when they’re completely immersed in that ecosystem. I just cannot learn it, though. And not for lack of trying.

Version Control

  • Git with GitKraken as frontend–it’s really nice–or command line

Messaging

  • Text message :wink: Phone call if necessary.

Misc

  • Bash shell… does this belong in “Misc”? Putting it in the miscellaneous category vastly understates it’s utility and doesn’t reflect the time I actually spend using it.
  • Keepass is indispensable password manager
  • Keybase, they’re really innovating in their space
  • Todoist–Could not live without this excellent cross-platform to-do list.
  • I run a local Postgres server (via Docker) and that’s been surprisingly useful to have on standby
  • Browse: Firefox & experimenting a little with Brave
  • Spotify web

Notes: I’m really struggling with notes. I was a OneNote fanboy for quite a while but that’s not part of my workflow any more. I love Evernote too but lack of a native Linux client is a bummer. Emacs/Orgmode–tried it (for months), I just can’t get into it. Plaintext or md files in a folder-- meh. Dropbox Paper? Has potential but is still somewhat immature for me. Anybody have alternate suggestions?


#6

I’m going to say it… This is by far the lamest, nerdiest thread in this forum I’ve seen yet.

And I still think it’s fascinating!

My setup is a hot mess because I really like playing around with different tech.

Work setup:

  • MacBook pro 15" maxed out
  • iterm2 + tmux + zsh (with ohmyzsh!) and caps lock remapped to Escape (not control, because I use control way less on Mac since Cmd key is more common)
  • I use RStudio for all things R, but in a few different ways
    • locally through the daily build on my own fork of the docker image (the 1.1.3xx versions are so good – full persistent terminal and fully dark theme!!! I <3 it)
    • on a fairly fat remote server on Google cloud for ETL stuff
    • on a modest remote server on Google cloud for Shiny apps (somehow running over two dozen Shiny apps on a single server with only 8gb of RAM)
  • sublime text 3 (finally out of beta, 6 years later) for lots of random text editing – everything from note taking, to documentation, to python or bash scripts, to markdown and making sweet, sweet ASCII art
  • VScode for Go
  • Pycharm for larger python projects (rare these days)
  • vim with awesome vimrc for quick text edits both local and remote
  • jupyter notebooks for helping co-workers with their python projects
  • sequelpro (queries and EDA) and MySQL workbench (tuning for performance) for MySQL stuff
  • pgadmin for PostgreSQL stuff
  • Google BigQuery through both web UI and CLI for data warehouse queries, but also for data processing where I just need to merge giant files or tons of database tables

Home setup:

  • xubuntu 16.04 (Ubuntu with xfce desktop environment, so clean, so lightweight, so fast, no bugs)
  • xfce terminal + tmux + zsh + vim
  • RStudio from daily release docker image
  • sublime text 3 for when I find working in vim to bothersome
  • jupyter notebooks with kernels for python 2 and 3, Go, Ruby, and bash for all the random stuff I get into (fwiw, jupyter + bash is surprisingly good tool for sharing/recording how to do complex system updates/upgrades/installs)

Media station (i.e. old computer hooked up to my big screen TV):

  • Dual booting Linux mint with kde and Peppermint OS (both are Ubuntu Linux derivatives, Linux mint is hugely popular and really bloated, Peppermint is a newer spin off of lubuntu and mixes elements from xfce and lxde Linux desktop environments, it’s pretty rad and really fast)
  • xfce terminal + tmux + zsh + vim
  • I shouldn’t be using this computer for analysis, but sometimes it’s hard to get off my couch and I can’t control when I find motivation/inspiration

I don’t think git/version control should count. It’s pretty much required at this point. If you use a special tool for it, I get that. I just use git as it comes from the terminal (sometimes use the RStudio git integration, though it’s still lacking).

I didn’t list any themes because I change mine every couple of months. There are just too many out there! I like dark themes with modest contrast (material dark, for example). I like monospaced sans serif fonts (really like hack).

I’m very reliant on terminal. It’s an amazing tool once you dive in and learn. It’s not difficult once you start. The difficult part is coming to realize can and dealing with the new found fact that you have a billion (not literally) programs running and available on your computer that you don’t see because they don’t have a graphical interface. I use tmux a lot for managing my terminal sessions (panes are awesome).

I try to use vim as much as possible for text editing, but it’s hard for me to find a good setup that beats the specialized options that apps like RStudio provide or the huge plugin system of sublime text. I use RStudio daily. It’s my favorite IDE I’ve ever used in any language and I credit it a lot for my improvement as a developer and a data scientist.

If you’re running Windows, I highly recommend using both a virtual machine running Linux (headless, no need for visual interface) and the gitbash command line tool. The Linux vm gives you a ton of options and let’s you develop locally just like you would on a remote server (unless your company runs Windows server, then sigh). The gitbash tool gives you a thin wrapper and a program called mingwin, which is like a mini version of Linux you can run on Windows. It’s nice because you can have some of the nice tools Linux does better (e.g. ssh, cut, bash, git, less, awk, sed) without actually needing to run Linux or have the VM up and running.


#7

I wasn’t going to post until I read @Ranae’s outline - but now I’m totally here for this!

OS
Depends on where I am and who I’m working for, so it rotates between macOS on a MacBook Pro, Windows 7 on my dinosaur of a Lenovo that I :heart: and you will pry it from my cold dead hands, or Windows 10 on a desktop I use for gaming 98% of the time.

Text editor
RStudio and sometimes Sublime Text

Command line
Terminal - until this thread I didn’t really realize there was more to it :upside_down_face:

Version control
GitHub, party of one!

Messaging
Email or meeting over coffee/lunch

Analysis notes
I keep things in an R project for myself, but almost everything I send out into the world right now is a PDF

Everything else
Trello boards - I swear by them, and they’ve replaced every to-do list app I’ve ever tried. Don’t @ me you Evernote fans :wink:


#8

:heavy_plus_sign::point_up_2:

OS

  • MBP 15" (maxed out bc Archer gifs demand RAM :woman_shrugging:)
  • Sierra (not the high version)
  • Occasionally my $75 Chromebook when on the go

Text editor/IDE

  • RStudio like 90% of the time…
  • Atom (which I’m falling out of love with)

Terminal

  • iTerm2 (currently trying micro editor on for size for terminal-based text editing)
  • zsh :heart:

Version Control

  • GitHub either in RStudio or command line
  • SourceTree and other Git frontends on my machine rarely touched

Messaging

  • Email, Slack, Twitter (sans apps other than Buffer…I don’t know why)

Analysis notes

  • Hardcore Rmd (and sometimes notebooks) user…like I write to myself as though I’ll forget everything.

:flushed: True Confessions/Everything Else

  • I write a lot of notes. This includes (but is not limited to): post-its, notebooks, sketchbooks, random pieces of paper, marginalia, mixed media and plenty of pictures :camera_flash: of all of the above.
  • Organizing the aforementioned items: sometimes I add indices to my notebooks, but because I often physically layer information, this is not an all-the-time thing.

Squad goals:

  • Integrating my ~free-form stuff with their more structured equivalents…recommendations welcome.

#9

micro is a cool project! Way better then nano, though some of the plugins are glitchy.


#10

This is such a cool thread! The environment in which more experienced folks work (or anybody for that matter) has not always been clear to me but I think there is a lot to learn in this space. Thanks for all the insights!

Ok. So now, my boring (n00b) answer:

OS
macOS at home. Windows 7 at work.

Text editor
Sometimes Text Wangler but now pretty much everything I do is on RStudio (see? boring.)

Command line
Terminal (when I have to)

Version control
GitHub

Messaging
Email, Skype

Analysis notes
Want to learn Rmd a bit better for this.

Everything else
Microsoft Office :grimacing: (is that ok?)


#11

@mara how do you find the notes when they are spread across such a diversity of media? I often write notes but have trouble finding them after the fact if they do not make it into my code.


#12

OS

  • MBP 15" (no touchbar version because the dongle life seemed too wild for me)
  • Siera
  • I have recently been toying with running ElementaryOs on VirtualBox when I want to rid myself of all distractions… I haven’t decided if this works or is just another distraction yet :grimacing:

Text editor

  • RStudio for anything and everything I can get away with doing in it
  • Vim otherwise (while constantly regretting switching to it and opening sublime but not loving that either)
  • For python projects I haven’t nailed a setup I love but end up using notebooks, something about them just doesn’t work as well for me as Rstudio does though :confused:

Command line

  • iTerm2 (now with the great dropdown from the top version thanks to @dylanjm :tada:
  • zsh (via ohmyzsh) I haven’t really personalized my setup much though, it seems intimidating. Anyone with good recommendations, I’m all :ear:

Version control

  • Github Desktop for commits on open source projects (i’m terrified of accidentally commit an extra file with keys or something so the visualization of it makes me less worried)
  • command line for work and private repos

Messaging

  • Slack

Analysis notes

  • Bullet journaling on a dot notebook (pen on paper is SOOOOOOO satisfying)
  • I really like Bear notes! Only for iPhone + MacOs though so if you have other devices in the mix it’s a no go. It supports synched markdown notes in such a beautiful way if you’re all apple products though.
  • I do ~95% of my non-package development R work in .Rmd

Everything else

  • Copy & paste from StackOverflow

@raybuhr I really like the idea of a jupyter notebook of bash for system setup type things. I might have to try this!


#13

I have a lot of notebooks, which makes it harder to lose stuff (including ones where you can reorder pages, which is nice). Post-its definitely play a role, and photos just in case. I cite my sources in my notes obsessively, too, because it’s often something I want to go take a closer look at.

I don’t so much write code by hand that would go directly into something I’m doing, but taking the time to pull things together helps solidify the knowledge.


#14

OS

  • MacBook Pro Fully loaded 15"
  • Sierra
  • VMWare to Windows or Linux if needed

Text Editor & IDEs

  • RStudio near 100% of the time
  • Rstudio Server on a personal server
  • Shiny Pro for App deployment
  • DataGrip for SQL (MSSQL mainly)
  • Atom
  • JuliaPro
  • Anaconda
  • Syzygy.ca (National Jupyter Cluster run by University of British Columbia)
  • Old fashioned Excel (not by choice).

Command line

  • iTerm2
  • Bash

Version control

  • git command line

Messaging/Planning

  • Slack
  • Trello
  • Alternote
  • Taskpaper

Writing/Citations/Misc

  • Zotero
  • Papers
  • Authorea
  • Rmarkdown
  • Atom
  • Baron Fig Notebooks & Notepads and Pigma Micron Pens (I am a sketchnoter)
  • Every cloud sharing service known to man
  • Alfred

Graphical Design Suite

  • Affinity Designer
  • Right Font

#15

OS

  • macOS on MB Air at work
  • ChromeOS on Asus Chromebook Flip at home running VMs

Text editor

  • macOS: RStudio / Sublime for non-R
  • ChromeOS: RStudio Server

Most final documentation is either Google docs, or R Markdown if using code.

Git

  • macOS: GitHub Desktop / Terminal if needed
  • ChromeOS: RStudio Server git panel or RStudio terminal

Files

Locally, Finder, but remote I skip Google Drive to the source, Google Cloud Storage :slight_smile:

Messaging

Slack, Twitter, iMessage

Analysis notes

R 70% of the time, python 10% when using AppEngine or GCP gcloud, 10% bash in VM Debain terminals.
Can’t live without Docker, have custom environments for code in Google Container Engine, call them down for specific tasks as needed. BigQuery a lot.

1password is cool to keep in synch across all machines. Use googleCloudStorageR and git to keep RStudio projects in synch along with config settings etc.


#16

OS:
Home: Windows 10
Work: Ubuntu

IDE:
RStudio (desktop and server)
Pycharm
Eclipse
Visual Studio

Text editor:
Sublime Text
Notepad ++
vim

Terminal:
Terminator on Ubuntu
Cmder (powered by Windows Bash)


#17

This seems like fun! :nerd_face:

OS

Main OS: Sierra (can’t beat the font smoothing on macOS)

  • Home: Windows 10/Arch Linux/Sierra on Intel desktop (might be time to move to the new Coffee Lake 6/12)
  • Work: Windows 10/Sierra on AMD Ryzen (had to see what R can do on 8 cores/16 threads)
  • Thinkpad: Work-issued; can’t do work on it…mainly a presentation device; if I really need to revise something, I Teamviewer home

IDE/Text editor

  • RStudio
  • Sublime Text with Material Theme (just couldn’t get into Atom; feels clunky compared to Sublime)
  • TeXShop (still working on moving my LaTeX templates to R Markdown)

Command line

  • Terminal
  • Power Shell

Version control

  • Git (command line)
  • GitHub Desktop (when I’m feeling lazy)

Messaging

  • Email
  • Slack

Analysis notes

  • R Markdown
  • Pen and paper

#18

Another for the simple life of setups, but mostly just because I’m getting started into my career and not sure of what I want to use yet, so I’m loving the suggestions.

OS:

  • Windows 10

IDE:

  • RStudio
  • PyCharm on occasion

Command Line:

  • Terminal

Version Control:

  • Lots of Files! I’m working on getting started on GitHub but I’m not quite there yet.

Messaging:

  • Email

Notes:

  • Word
  • RStudio
  • Lots of Notebooks
  • Google Keep for lists

Other:

  • Spotify or Pandora for tunes
  • Microsoft Office for a lot
  • SQL Server
  • Google Chrome

Edit: Added Bullet Points to make it look pretty :slight_smile:


#19

Really? Only a couple Excel users here? I know there is a lot of justified dislike of untidy Excel tabs, and I can relate to not wanting to pay for it, but how do you people edit a csv file? I personally end up creating lots of hand-made crosswalk tables in Excel, and I searched through this whole thread trying to figure out what other people use instead without any luck.

OS:

  • Windows 10

IDE:

  • RStudio

Command Line:

  • I basically don’t use command line, except a tiny bit of git bash and Powershell

Version control:

  • Git / Github / GitKraken
  • Because I use GitKraken, I have not yet had to learn command line git

Messaging:

  • Email, cube visits

#20

Don’t worry,you aren’t the only Excel user, I stuck it in the other section under Microsoft Office. I use it for a lot of different things, but the most common is data entry from paper surveys or similar.