Maximum Total Header Size ~8KB?

shiny
1

Hi,

When we send a request to shiny server and the HTTP headers get to about 8KB, the shiny server will "hang" and not connect. After 45 seconds we get sent a FIN packet.

If I add 1 additional character to either my-header-1 or my-header-2, the shiny server will hang.

GET /dashboard/ HTTP/1.1
my-header-1: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam sed dolor ut neque semper fermentum. Aliquam erat volutpat. Nulla molestie euismod magna, tincidunt tempor mauris tristique semper. Integer vulputate, ligula ac pulvinar fermentum, massa massa luctus leo, et sagittis augue risus nec quam. Vestibulum commodo tellus pulvinar metus ultricies, a viverra sem vehicula. Duis ex nisl, semper eu hendrerit quis, molestie quis est. Nam commodo ut lacus et dignissim. Praesent et nulla metus. Pellentesque id turpis ullamcorper, pulvinar ante ut, malesuada nisi. Morbi molestie eget nunc eu ultricies.Phasellus elit eros, ullamcorper id ex sed, pulvinar auctor ante. Cras sit amet nunc erat. Nullam a magna et massa ultricies volutpat ac nec eros. Suspendisse potenti. Integer in magna sodales, euismod mauris consectetur, vulputate massa. Mauris euismod eros sapien, vel suscipit erat elementum at. Vestibulum scelerisque, ante ut fringilla feugiat, orci orci convallis velit, sit amet feugiat ante neque quis sapien. Ut massa risus, sollicitudin non eleifend eu, tempor id ligula.Nulla iaculis massa eu dictum dapibus. Aenean et blandit odio, in consectetur felis. Quisque eget urna quis justo molestie consectetur a sit amet urna. Cras ornare porta neque sit amet hendrerit. Lorem ipsum dolor sit amet, consectetur adipiscing elit. In nisi quam, interdum ut consequat vel, bibendum vitae velit. Sed commodo libero in luctus porttitor. Nulla nibh orci, dictum vulputate velit sodales, bibendum congue nulla. Integer eget feugiat lectus. Praesent aliquam mauris ut magna semper egestas. Sed nec ultrices tortor. Vestibulum dignissim ut metus in commodo.Nam turpis erat, consectetur nec tempor a, ultrices vel dui. Cras et tortor tellus. Curabitur enim diam, semper et pellentesque nec, placerat imperdiet metus. Sed faucibus odio risus, et aliquam justo placerat quis. Praesent nec lacus lorem. Aenean elementum lacus arcu, nec posuere lectus malesuada in. Sed laoreet cursus leo, et congue eros blandit in. Vivamus bibendum, magna iaculis hendrerit lobortis, erat augue pretium purus, vehicula sagittis neque massa sit amet nunc. Quisque quis elit vel libero semper elementum nec a enim. In iaculis semper libero in laoreet. Vivamus egestas quis tortor imperdiet tristique. Sed quis sagittis nunc, at viverra felis. Nam blandit, leo eu porttitor placerat, erat velit aliquam diam, non fringilla eros mi non erat. Nunc ac diam eu lectus pulvinar sodales at vitae nunc.Integer dignissim nibh sit amet pharetra ullamcorper. Phasellus non nisl velit. Praesent quis dignissim ligula. Sed non volutpat ligula. Proin urna arcu, ultricies eget sodales at, pharetra aliquam elit. Proin sagittis euismod mi eget scelerisque. Sed id vestibulum massa. Etiam malesuada sodales ex, at fermentum libero pretium quis. Proin quis rhoncus turpis. Fusce nec nibh mattis, malesuada ante pharetra, rutrum lectus. Integer eu tortor purus. Vestibulum pretium nulla molestie, luctus eros ut, accumsan turpis. Fusce quis diam hendrerit, congue metus at, rutrum neque. Cras faucibus id odio eget rhoncus. Sed eu pellentesque ante. Curabitur nisl quam, aliquam nec pharetra vitae, venenatis vitae sapien.Duis vitae consequat magna. Quisque et vestibulum est. Mauris et erat mauris. Sed eget molestie elit. Nulla facilisi. Suspendisse accumsan ullamcorper pharetra. Pellentesque quis elit sit amet nisl luctus volutpat non nec nunc. Nulla vel leo ut erat posuere placerat. Vivamus eget risus euismod, elementum quam non, fringilla urna. Donec eu lacus non mauris faucibus efficitur. Duis cursus ante lorem, a cursus nibh consequat vel.Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Sed feugiat scelerisque ullamcorper. Aenean convallis nisl id neque convallis tempor. Aenean vulputate turpis et nulla venenatis, eu vestibulum erat dignissim. Sed faucibus, odio sit amet fringilla ornare, neque velit eleifend urna, quis ultricies nibh ligula ac lorem. Nullam non mi quis nulla viverra pretium eu eget massa. Sed id tincidunt ipsum. Nam vehicula metus quis enim malesuada, nec venenatis lorem pretium. Donec nec mauris et ex facilisis commodo sit amet sed turpis. Morbi a felis in odio pulvinar sollicitudin.Fusce ut finibus magna. In malesuada facilisis luctus. Pellentesque velit dolor, faucibus quis semper eget, tincidunt vitae lectus. Donec pharetra mollis quam, quis vestibulum orci blandit a. Nulla gravida dictum odio, at ornare metus sodales id. Vestibulum sit amet scelerisque est. Proin sollicitudin orci ac tellus fringilla, nec ultricies tellus pharetra. Mauris at sapien sit amet turpis varius eleifend. Donec vel sem faucibus, eleifend risus nec, faucibus ante. Nulla convallis non massa maximus placerat. Aenean dignissim tincidunt orci, tincidunt tempor turpis fermentum et. Proin maximus auctor nunc, non efficitur eros cursus et. Aliquam ut dignissim orci. Vestibulum molestie lorem lacus, sit amet posuere lacus vulputate vel.Aliquam vel libero quam. Nullam dui urna, vehicula a vestibulum suscipit, sodales sed metus. Sed gravida, arcu ut congue tincidunt, magna felis sagittis nunc, ut semper lorem metus et augue. Suspendisse malesuada, magna non ornare sagittis, sapien libero dignissim sapien, a cursus est nibh non eros. Maecenas vehicula massa et nibh euismod tristique in porttitor lacus. Maecenas eget rhoncus felis. Nullam ullamcorper purus massa, quis feugiat sapien fermentum eu. Sed eget turpis vitae magna vehicula molestie at nec turpis. In et nunc non tortor aliquam tincidunt a quis sem. Aenean gravida tincidunt euismod. Phasellus quis fringilla tortor, eu ornare risus. Quisque non sodales ipsum. Quisque auctor nibh at ex blandit, sed feugiat lorem cursus.Donec posuere aliquam leo, eu tincidunt turpis porttitor porttitor. Donec at condimentum mauris. Maecenas enim elit, facilisis eget sodales non, commodo nec mauris. Vivamus vehicula lorem quis diam pharetra, sit amet iaculis erat egestas. Quisque suscipit fringilla mollis. Aenean ac lacinia arcu. Vivamus at pellentesque arcu. Nulla nec massa varius, tempus urna in, elementum diam. Nulla feugiat ex sed purus hendrerit, eu tincidunt ex tristique. Sed consequat pharetra lobortis. Nullam nec nulla non ante placerat ultrices. Duis non nunc ullamcorper, convallis tellus a, placerat dui. Morbi eros nibh, ullamcorper ut diam in, vestibulum placerat arcu. Pellentesque auctor orci sed sem bibendum fringilla. Sed vitae lobortis lectus.Ut commodo id lectus id dictum. Nulla finibus, velit sit amet molestie mollis, orci mauris rutrum lorem, a semper sapien nibh vel ante. Maecenas venenatis eros nec rhoncus ullamcorper. Cras a fermentum enim. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. In sollicitudin nibh eros, non pretium nunc malesuada ut. Praesent venenatis turpis lectus, vitae luctus est pretium in. Donec posuere dolor sed dolor molestie, quis varius odio rutrum. Nam elementum ipsum eu nibh posuere volutpat. Vivamus feugiat id aenean.Generated 11 paragraphs, 1043 words, 7000 bytes of Lorem Ipsum
my-header-2: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam sed dolor ut neque semper fermentum. Aliquam erat volutpat. Nulla molestie euismod magna, tincidunt tempor mauris tristique semper. Integer vulputate, ligula ac pulvinar fermentum, massa massa luctus leo, et sagittis augue risus nec quam. Vestibulum commodo tellus pulvinar metus ultricies, a viverra sem vehicula. Duis ex nisl, semper eu hendrerit quis, molestie quis est. Nam commodo ut lacus et dignissim. Praesent et nulla metus. Pellentesque id turpis ullamcorper, pulvinar ante ut, malesuada nisi. Morbi molestie eget nunc eu ultricies.Phasellus elit eros, ullamcorper id ex sed, pulvinar auctor ante. Cras sit amet nunc 01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678
User-Agent: PostmanRuntime/7.24.1
Accept: /
Cache-Control: no-cache
Postman-Token: a236f54b-6ab7-4125-abbb-3c3fec3b4606
Host: devintshiny02.somedoma.com:3838
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

2

Size limits

The standard imposes no limits to the size of each header field name or value, or to the number of fields. However, most servers, clients, and proxy software impose some limits for practical and security reasons. For example, the Apache 2.3 server by default limits the size of each field to 8,190 bytes, and there can be at most 100 header fields in a single request.

Wiki

3

Thanks for the reply!

  1. None of the headers is over 7100 bytes. Please see that from the provided request.
  2. This request is direct to the shiny server (no proxy) - is the Shiny server a version of Apache?
  3. You can see from my request we are not close to 100 "fields" if you are counting headers as fields.
    And the request has no query string or form parameters.
  4. Note from my posting that adding a character to either of the my-... headers causes the issue as well. You can see from the 2nd header is much smaller than the first ~7000 character header.

The request data above is captured via Wireshark and is exactly what was sent into the Shiny server

Given this do your suggestions help explain the issue?

4

Hi hedrick, sorry for your trouble, it must be frustrating to come up with these unexpected issues.
I see that the example you sent is on the order of 7k and 1k respectively. I was drawn to your opening comment that you empirically noticed issues as you ' get to about 8KB'.

I don't know if shinyserver is based on any other web server technollogy, I referenced Apachae as an example, I've searched around and seen other servers with limits of 4k, and others where the admin can configure it. I read that when the limit is reached, servers should respond 413

shiny server is open source, so you could raise an issue on the github, and probably get a much more technically proficient response than what I can tell you as a mere enthusiast :smiley:

From my admittedly limited standpoint, the approach I would be to inclined to take is that http headers are a resource best kept short to the minimal bytes, and to keep them lean, and put significant data objects in the body ? 1k does seem short though ! sorry for not being more helpful.

5

I appreciate your time nirgrahamuk! I will leave this open to see if others can help.

I will likely head over to github too. I don't usually start there as I hate to pull from development time if there is already a user community member that is knowledgeable of the issue.

6

Why are you trying to send so much data in an http header?

7

Building on @hadley's and @nirgrahamuk's comments: if you have that much data to send, the way to do it is with a PUT or POST request, with the data in the body.

8

Hi everyone,

Again, I appreciate the suggestions. I do understand HTTP, the protocol, and all the nuances between GETs, POSTs, headers, and BODY data. While the suggestions are helpful they are missing the question.

What you don't know because I am trying to keep the question simple is that there is a front-end system that proxies requests to Shiny. This system includes data in headers from a SSO system. This allows the back-end systems to not have to process authentication. The sso system then passes the needed data on to the back end application, Shiny, to implement authorization. I intentionally created a test case that leaves all of that out to prevent many well-intended questions about multiple software tools and infrastructure that often lead all of us down rabbit holes!

As you can see while all of your suggestion work around an unknown Shiny limitation by you as well as me, these suggestions just aren't workable for my situation.

I understand how these limitations get put in place. After all, I have had to solve this same issue with F5, Apache, Tomcat, Php, and Python based servers. Shiny is just a new one for me to have to work on the issue.

To help anyone else that runs into this issue and is looking for confirmation of the problem as well as a potential work-around, please see a thread I created on GitHub. Here the issue has been acknowledge and a potential solution has been posted.

9

Thats great that Joe Cheng could help you. :slight_smile:

10

I was able to solve the issue by adding an environment variable called NODE_OPTIONS to the service unit. All details are found in the previously mentioned github issue.

11

This topic was automatically closed 54 days after the last reply. New replies are no longer allowed.