Does Console impose an upper limit on the length of strings

james_hedges · August 20, 2018, 9:06pm

Trying to create a string of length 5k in Console results in a next line prompt ("+") wherein another closing " needs to be added even though one was already entered. The resultant string has length 4087.

Is 4087 the maximum length that can be taken as input from Console? Is this documented somewhere? This is well below the default maximum length of string values (i.e., 65535).

foo <- "swY4Dura4GSPHlLTgJgvNDkRmCtMd8J3u6tvCkwpltCX4RtTkxwg0GpftrdgI1YAnF6ykAr3gkNsq9oawBOeh7VYVZnAIYILFa0eT8dY1zxv88SWM3cOfQ6jXZtWqybqZu1xelmRIIudOUkIIzALyE9oen3JFCuvLB4UFkqRDfukVRIo2wuR0fEPN6GmyAzfMiPKCRTpuRDZGxWz82rTqbXwHmL7Z3qOgqbhjXU2rYDqceze80WgNWvOK3nyixWUPebmdX8k27lynityv3eQtFRo9gjegECsvPZRBg3QerwqMZw5etBFYuxR5onCG6ES6QkVW5joqM5yEMgwTIcNyiPFp6UMArmIe5YDsrTnCH1Pw0WJigbOClVZPut4sYRybWblyXefPGRo9k1yZwJLXKwRbquVQrHOVKuxCMTJv9LPu4ftv9vrDd6RsDUYEAcQerZjNDu3PyBCu9gnmThSMQo47odNMTVLi8Zz9jRWHQdTav9lgWTkMOty7878sxLeDRC8RBUjvi8fSgCXnV2pX2J2BeohDIYXTjzn4HzaA5JZUIefcwxdvqylnVbYs94YrAgQ7e0KNpJuCVzvnrzBZvAML2N28Wf2lEvAgH4KDjOoTelyQyJZfT9NA6Fqv6iS4sVs6PGXGVuBxoXZtLbmQ07stvksYCJKu8SP59fFHSBoCOS5sBgqsM0Q4OX6Eu5Vf9yOAi1V0I6oBgjNSzWAJYuipIuRSCvz538kAx0ULIp3zlqti8X1iErwx4ciXmcpUB3zYXfnypUAbWyTwJnUsV22TUDAtIrCJ6Lz9xbhnQem6MfPg44gb4ByQm9jJJXC0YvL4nlX69vFRCuiOqq5uNzJO8nulu8a6b9oHPqaZydk5TTocpCrdgvlATa7hzJOl0TahqVAwaRupvcgVjpEaWbbY51EiuEUruHMYcjyF6tNPigR1KayrsxKJFTYqzfZm4CnnhrNNCsbc4wKSlEUUPSRYCtR3UH1DQ5vvcYMigdTulugwkkqZasnoDipPbyrxTxrb9Cvk8hVZ4NoY1wVYY5DpVGXCSemrCyX7RVe9SC5hjcmlSP6JadhlwhvCvTOmJPQzDvmg4gl6pF1mG7ek67tqpCabyQ6GGXFEqMiWCVLxYOFTHSgfr6mB52VERCFJ1HbQBIqd82OJkj5IEzN15TZ9fYatTozwXG7H8RwnJr9Td0pNyjNOFcNkQWRWNkyMRKoopdPuFxOT6kDC2eWY17nM8pIFLoz6YnZS1AtdfAQLBIFJCLiHQfUe2vYhZus9lcIut7aUMCYhVPPn4oM8rA6kfGluLyV6AEDA0Sc1gjAGS4CZmMQ4kodzk6AxSwiYkyHxZ1Q3jpn6JUTm7RDvPvQEYhf2LFhxWBllHXyCHqmCZWOkYXhUDAiAk0u0ReTE63dGb0qTMP9E1UpcCd6SPVSJP2I1TaMm3TtUM90eiNbFGqpxQ9SRv4adUMjv7YcNBs2b9nBzyF70Ow0N0S6uMwZrgazPTPZJ7bILn4PYC5a0sbLdGNZBucRwnwarJhmg2RnWCgvbQWG1O4NeJCzqkHCbHMzUR4oY4mDokrbkxQu6Q0jE3AZF9yjIKNSaN2Hbo1MvCqvqmoD90W0Zih1yXp8J6NYKiajyQbiyYsVGwKZqXi6A7N4Pumk6cS96Gz0F5HSwjgV7YGPvfgYjC2pkq56pHj6grPCQvHJBpy01L0hoPv3Gh4P5LH4YRPS61JvuKWLUc7qUntkG8VOu4DChTbNFpmhTVlJUUh0xl26OehcryD6m855y4AHAUQL6bSqTdIHtUU3FWcOC8je6VFiYTfgFoIvzhuWgwTKu00q6aF9XRQ8kKCnsEQ1v9u0d2v6faZpRxh1oIqM9aLvzLMpDvf5MaKtplksIOe4iwySM69JI1741LNBfIN4zxRUEyrdNTkBNTgJRD0HLiMhjU8V3EVNKoDZwXYbfFGiKuvyoPCx7OH3i1pJcCjTfw7gdn2pSUR6QFU3qlFSS8cRPKVtDwZd14ftfd9Ls6hMo2GYoEg64NvJUvUpCHCz3QZDFlx00f2rZuvL1JcePAUPLJWoMPwD3ZC0SWeUbcQrNaanS2QFoOTgpMNA9AIIVM2pf83EXmMk3qukw47vke0S7bMQZrx8WYcEUM6JLtSnncLojufTlFyP5kPAJCtL3qrHfObmPXMXKbP4xTzDiyDhNVyLzLideIFqunVIYnzPmkitXvXrldHvXY6TpzYMWO8KElhpguXDWx9f30ZQ5soXza0hEGqgtjLDZQ3XNzN1d2u5NOAgXwN81oXm1IkR5j6b0ZmdnIGyfEL13qLEvamuydn2O1G1KbzhURfpvH6sifH0XCygxyvrKPMYsibTGLSkCsMTwTcn7IkGu1U1oB5e8t08aifWAyf5m8uBKdPquBL0j0C85MUr5HKaM9K8EEJfrgtAISOii9Zmt9c6R2MQPo4rV88vXqtYUtbVSRLzldwHHT5ZJk2tSyfrUbH0q8YgbTlprp7SeCICLztDXx2N5d0iNjCfyd0PZincvSs4XBHzFNM6lr3yzK3E8llOBmlKNBYFvDD45QoSM2pSHN5norbLuxOXXlFKFaCMVVATKsqYaPjGfwKxLM10hPAifSHku9NODc2zxtY4qDJYmnPbiSrN7OnFGPUE42JofIsVRj5G0YSDwcz1e9autnBixsULhaUkJwzb4AQ4qumQjVeaaCqg8faboDIl6rbybpB1ZizPoUNFpPQaybQYRcIH7NeY9q6IjUSHSp18hOHuiG3SbTuUOaKNhKxe0gqGBYKS8AE5GemDVrlZSFLBftdQTS4RW57egMZrrEwEXZN5D2jwS9rXnHOPEo436zU7GlhJ9iXKLmgrVCL7mb2CmneY2SMRWAeEesB0FXTTFSNZDWimXxIBSTTZG40pB7XBvBDFrcmvVO1VRLAiYFudH8AUYAgFImEyL5MldvSxPMSKY5wGnaanBlGWBR9kgzpvWZ4xs7lg3imWrC5CvZ8VCdT3c4c9m95r8IZXuR7XBOV6KdJ8H4jVd00jU0UiVjXzcQ4ZPxVBJUuYbhLDVU6pWexpWthYWhv6NItaanJEv73Dg6tUyz6lJdw7pKdZcibGZUZXyz7v6nFm2MNHOBE4XU0IcXERNPaHZUazniadgF0IpyokrANpYEgyMw5bzCftDcUq2BIch1wgx5rRbTAABGx0vipmMwgubATSdK7Xnpimy9SRjVi71HOiK2ZtToFoIpVvCmuhejRkSZl99ySvQFfCPzS5UnrjweuCPwUTYUOv2jviFXr4gKxctNNf5z9OhiaVhZpDqMtQPhGiYAyGJiErDo40IP9poqzdJHPM5rH6oYP4K2eq4ULRYLIvnMEZSv8KcMQDDSDp5aH4F0vRBE7D7jeQP9dzt2a1NmNXPDSIJ3jMraoVAvSorCt7WdVAt3SDrVy7AgY0WBLYvYBNQpvhhHw69t2Mb6wNBggvcYegoICIzXXlK1W8ML2JmecCME7q48Bx53tn6fO90uJxqAXOLjBwhdN5SBHGBCFs2rmzgF8oqIL3PqQhDPgYzs6LTK4zqBc14iXyMwsrdXrZY4bRrJ4ae2mTPphkT4wTKlQkRW37FFPEJWqyQF48lTsig5ssShP3OSQQOYo0fCiLU5IEl7lWT3dtiHUiwaPFiMQUVRqcKzU3JhwOB4klAUZvTaOC5HUfv5EbnrdKdEMFXH2z7zYFZoI5Sk5PhDMx8ytURtp1qzq9rhoMEx8mZM5Lhc2k6nC1FB8tJQP4IQ4PrrPtUCCSLMzSprceyf0myy8ZoyU3v4PCERlNpZBwkLfh7WCcCVcyw3fuLE8Hc0PU9M55pERqj8VT9Jz1gXZDlZneQSDrvav9uDwKisKvO4WRhKWJlodKpiBnzgFXeCXSrHXnVGTeasfLBdJJuPOOVhkpV0bm8u6PijlPCCZvdEeoeOmQiSzC9FD6FlMXZ0ftbutHF1NKQ0O635QMSX4GcAyhlcqV91pGr1ic10ZCi1T0NyQGwc2vDEK9lQQLAXUR3hpc1ktKVZGafKFuZjXmoURiB0GsUWHTnkS8Po23bX34zb99xFNUHK0DsIAcsbSSR2aKEbsdyfQvfUDOCTdTV9D8ycQLnncuP1PJwSqu3dc0fQQkfPZQqRKrH1D72RrlVJCNlOWzWjWT34BaznzGV3sfbzU3lw0axYvJO9zNRBSFSFayJloMqnPaYSOuo3WxXW77nDFMqt20boDFRsRubs0CuPXLJGgBGw8Neyt3PnZIAAGIhkSYZ7jdbJVhgSq55hmiWplTJCavwABYARDG7n8JoDrq2b8JJNyNAI9kzQPe85uJs8guSJoMp5eAgd4lD64bqUDV3MtKoUWvSoteLbvVHC3JvIHcsJmdKJ1BIFE7zECcF5ivDa1ddYyct9E21JhL8QqRTqAsWXlz7BZtVuyCvlrH6IRsPW2if5Q58MeD0cwrV0KaA25O6Adp2VZCURLmkgaw4hMPqB8zkMNGa05yaOc2sE0fLuWvrIOy8MuTzlhtHzuwcJC5mtJkSVrw1HnecQKChRRLu38w2LshK4o6dkMUzle9Tiu3lvIcSaBeby3HIrxtY3NBbpIdLIvauU30PuAp0TKZSztbwHOIsO1OXAbSp94xflpcWHWAyRNDLG0iDKHf9AVcKYD57WupoKS5qUMHw5feaJHFR4hEDUZrSDMOfmIfXE4JlSQn9dG4AfcmeRtWShWg7ffp4No93Jde2NperQGQvd9kRJrPkrdF1jIcWzc2q1IL1BnDRSbehUM9ETRzLXpqKXrhykUBlexqTblsoP9vCYwX1ha0bPoJ69qxq1zbUVdCV3WjRqJXT2YpNYR0alMVgnSpVLE6hRkAxfT1DYhzqraRihq0C2Te1bb5oXAkz8stiBV6S9GYMGB8QUKDScr3MEmP7h8R3IgaoZSrKDuLg1o2R9whTB11V0UKmHuwY7uCjEH1f3fuTXfCrWaOaS77vfF0Tp97857BHNWmQooSfTPKTMC5"

jcblum · August 20, 2018, 10:22pm

Just to clarify: is the behavior you describe only present in RStudio’s R Console, or do you see the same thing outside of RStudio (e.g. via the R GUI or running R at the command line?)

mishabalyasin · August 21, 2018, 8:10am

I seem to remember having the same problem and, considering how close your number to multiple of 2 (4096), it looks like it's the same problem.

When I had the same problem I think I've found that this behavior is configurable, but for that you need to recompile R with correct flag set to a bigger number.

cderv · August 21, 2018, 9:10am

This is a limit of the console. it is mentioned in R documentation

Command lines entered at the console are limited4 to about 4095 bytes (not characters).

It seems to be hard coded in the source code

github.com

wch/r-source/blob/496adb0085703362b2dee63537f0d89820f5afbb/src/include/Defn.h#L1040-L1042


      
          /* The maximum length of input line which will be asked for,
             in bytes, including the terminator */
          #define CONSOLE_BUFFER_SIZE 4096

You can circumvent this by creating your string in a script then sourcing it or by creating it by pieces and pasting all in last step.

Example of creating programmatically a 5000 length string

test <- paste0(sample(letters, size = 5000, replace = TRUE), collapse = "")
stringr::str_length(test)
#> [1] 5000

jcblum · August 21, 2018, 1:47pm

Since this sounds like an R problem, not an RStudio IDE problem, I’m moving this post to cut down on noise in the #rstudio-ide category.

james_hedges · August 21, 2018, 2:06pm

Hi. I checked that quickly before. From what I can tell R.app does not show it, at least not in the same way (i.e., it doesn't seem to have this issue). R in Terminal does show it.

From comments below, it seems like a property of console. I don't really get 'the why' of this yet, why that constraint would be built in.

james_hedges · August 21, 2018, 2:08pm

Cool. Thanks. Sorry for putting it in the wrong place initially.

james_hedges · August 21, 2018, 2:09pm

Awesome. Thanks very much.

Would you be able to say why that is hard coded? What would the problem be with entering things greater than the 4096 limit?

jcblum · August 21, 2018, 2:35pm

Spelunking in the source code, I see that 12 years ago, a previously undocumented and un-centralized console buffer limit was formalized (at 1024 bytes), apparently to deal with a Windows FileReadConsole problem:

Then two years later, the limit was increased:

So for a “why”, I’m going to generically guess good ol’ “performance” . For more historical detail, I suspect someone would have to ask Brian Ripley.

ETA: Because I love a good research project... there is a bit of kerfuffle around this topic on the r-devel mailing list in April and May 2008 (linking to the end of the somewhat contentious thread): https://stat.ethz.ch/pipermail/r-devel/2008-May/049553.html Doesn’t really answer the question, though.

james_hedges · August 22, 2018, 6:56pm

Thanks! That's helpful.

Pretty weird to me to see the perpetuation of a kludgy and localized fix like that. I could maybe see aligning such a limit to the one imposed on strings, but having one that falls well below that doesn't make sense to me. If anything, it seems like it would be helpful to have an optional setting of that kind for users who are affected.

Cool to see that discussion, although from what I can tell, it doesn't seem centered on this issue, or 'the why' of the issue. Good to see Dirk there. It actually seems like the kind of thing Rolf would engage on, battles around form, whether a reproducible example was provided, etc.

jcblum · August 22, 2018, 9:58pm

Aha! I found the origin story for CONSOLE_BUFFER_LENGTH:
http://r.789695.n4.nabble.com/Command-line-length-limits-in-R-tp912367.html

(got easier when I remembered about the Nabble mirror to the archives, which is searchable)

Not sure this throws any light on whether other solutions were considered before the limit was raised (or since then). It’s a little tough to search for this topic since there is a lot of interference from discussions of the “vertical” buffer limit and line-wrapping, among other things.