Friday, June 28, 2019

Workaround: HTTP 502 "Incomplete response received from application" in Rails app on Phusion Passenger server

I've recently been fighting an intermittent issue in a specific Ruby on Rails app running on a Phusion Passenger web server on my local macOS development machine: HTTP requests to the app would intermittently return the error "Incomplete response received from application" wrapped in a simple h2 element, instead of the desired resource.

When this occurred, I'd also see a message like the following appear in the Phusion Passenger web server log:

App 6510 output: [ 2019-06-27 12:07:49.0318 6510/0x00007f83639eb6f8(HTTP helper worker) utils.rb ]: *** Exception Errno::EBADF (Bad file descriptor) (process 6510, thread 0x00007f83639eb6f8(HTTP helper worker))

Oddly, no one else working on this application saw these errors while running the app on their machines, even though as far as we could tell, we were all running pretty much the exact same environment.

The workaround

 I found that I could get this issue to stop manifesting on my machine by appending the following parameter to my passenger server start-up command:

--spawn-method direct

This resulted in the complete Passenger start command for the application looking like:

bundle exec passenger start --port 3000 --max-pool-size 4 --min-instances 4 --spawn-method direct

Explanation

The key to coming up with this workaround was an article on the Phusion Passenger website: Spawn methods explained (for Ruby developers).

That article talks about how, in order to save memory and improve start-up time, Passenger uses a "spawn method" setting of "smart" for Ruby applications by default. That setting accomplishes that by having multiple server processes share certain objects in memory.

A caveat of the "smart" spawn mode noted by the article is unintentional file descriptor sharing. This caught my attention: Research I'd done on the "EBADF (Bad file descriptor)" error noted that it could be caused when a file descriptor -- that is, a handle pointing to a resource like a local file or a network socket -- is closed, but then accessed again after being closed. If such a handle was being unintentionally shared between web server processes, it does seem entirely plausible that one process could close a handle, then the other process could try to use it for something.

It seemed logical that configuring Passenger to not share objects between processes, via using the "direct" spawn mode setting instead of the "smart" setting, would avoid this. Sure enough, using the "direct" setting did immediately stop the "Incomplete response from application" errors I was seeing in my local environment.  Additionally, running the app on my local machine (with myself as the only user), I didn't observe any meaningful degradation in app performance.

What I'm still unsure about at this point is exactly what specific handle was being incorrectly shared between processes, and why no one else working on this application was having the same error that I was manifest in their environments. Thus, I'm considering this a "workaround" rather than a "solution" for the time being.  (And I'm not committing a change to have the app use the "direct" spawn mode in any environment other than my own.)


Other solutions attempted

Prior to finding the workaround, as all signs pointed to this being some kind of environmental issue with my local machine, I tried to "fix" my environment in quite a few different ways:
  • Use a different client browser
  • Restart the Passenger web server
  • Reboot 
  • Reinstall Ruby 2.6.3
  • Revert to an older Ruby version (2.5.3) using rvm (and uninstall 2.6.3)
  • Reinstall gems via bundle install --force
  • Clone a fresh copy of the Rails application code itself from the remote repository
  • Run the application on a different port (other than 3000)
  • Update the database (mySQL) to the latest patch version
  • Increase the limit of open files per process via ulimit -n 8192
  • Increase the database pool size in database.yml
None of those attempted fixes made a difference; the "Incomplete response received from application" response to HTTP requests, and the "Errno::EBADF (Bad file descriptor)" in the server log continued to manifest.

I had initially suspected the Ruby 2.6.3 upgrade to have something to do with the problem, as I had first started noticing the issue around the time I made that upgrade locally. However, as downgrading to 2.5.3 didn't fix the issue -- and others working on this app were running 2.6.3 with no issues -- I ended up concluding that the 2.6.3 upgrade did not seem to be the direct cause of the issue.

I'd very much like to understand what is actually causing this error in my environment. In the meantime, though, I'm very happy to finally have a viable workaround, so I can get work done again!

Wednesday, February 20, 2019

Throttling Skype bandwidth on macOS

I'm a full-time remote worker, with all of my teammates located together in the office. To help maintain close communication with my team, I have a "telepresence" Skype call with the office that's always on, so my teammates and I can see and talk to one another.

We're currently using Microsoft's Skype as our videoconferencing solution, as it is the only one I've found that (1) works reasonably reliably; (2) supports auto-answer (so I can activate the call without someone in the office needing to manually answer it every day); and (3) is free.

Unfortunately, as with many Americans -- particularly in localities where only a single company provides high-speed land-line Internet -- my Internet service provider recently started imposing a bandwidth cap on my monthly Internet usage. My cap is 1000 GB per month. Between my job, and my family's normal Internet usage, we've been either coming close to, or exceeding, that cap on a regular basis.

As one part of a strategy to try and keep my home's Internet usage under the cap, I looked into how much bandwidth that always-on Skype call was using. Using the Network tab of my macOS copy of Activity Monitor, it turns out that it was quite a bit: An average of around 350 KB per second (with peaks over 400 KB/sec).


Over the course of the typical 6 hours per day where my team (Pacific Time / GMT -8) and I (Eastern Time / GMT -5) are both online, that works out to about 7.5 GB per day; which in turn works out to about 150 GB over a 4-week period where I'm in the office 5 days per week.

Unfortunately, macOS 10.14 Mojave does not provide an out-of-the-box a way to limit the bandwidth used by a particular application.

Skype for Mac (version 8.39) itself also doesn't provide a quality slider or other rate limiter; it always appears to consume as much as it can.

Complicating matters, Skype 8 also doesn't specify or allow configuration of which specific TCP ports it uses, for possible QoS throttling at the router.  Per a page on the Skype support site, Skype might be using TCP ports anywhere in the range 1000 through 65000.

After a fair amount of research and dead ends, the working solution that I finally landed on was the free version of a software package named Murus, which provides a GUI wrapper around the Packet Filter firewall functionality that comes with macOS.

My Murus configuration is based on a very helpful post on the Murus forum by user "hany".  The variant of hany's instructions that I used is as follows:

  1. Install Murus.
  2. In the main Murus window, in the "Services Library" pane, add a new custom service named "Skype" (or whatever you like). 
  3. In the Ports field, enter 10000:22465, and then on a new line, 22467:65000.  (I excluded port 22466 because per Slack Support, that port is used for Slack calls, and I wanted those to remain highest quality.)
  1. Drag the new service to the "Managed Inbound Services" pane.
  2. On your new Skype service, click the little speedometer-looking icon to bring up the Bandwidth dialog.
  3. Set your desired values for the Upload Bandwidth and Download Bandwidth. I found that values of 768 Kb/second resulted in very usable (if slightly blurry) Skype calls.
  1. Press Cmd-S (or select Firewall > Save Configuration from the top menu).
  2. Click the Play icon button in the top toolbar of the Murus window to start limiting bandwidth. (Press the Stop icon button again later to turn the throttling off again.)
  3. I have found that I sometimes need to quit and restart Skype to see the changes take effect, while observing the bandwidth usage in Activity Monitor.
This is admittedly a non-ideal, somewhat brute force solution, since it affects not only Skype, but any application running on the computer using TCP or UDP ports in the 10000+ range.

However, it has proven to be effective: With this Murus configuration running, Activity Monitor shows my incoming (download) Skype bandwidth dropped to around 80 KB/sec.


That works out to about 34.5 GB of usage over a typical 6-hour-day, 20-day work month -- a savings of some 115 GB over running with no throttling over the course of the month, representing an 12% or so usage reduction in my imposed 1000 GB/month cap.  Not incredible, but not terrible, either!

Thank you to the Murus team for the software; to "hany" for the posted on the Murus forum that helped me with this solution... and to my local ISP monopoly for the opportunity to undertake this interesting learning experience!

Wednesday, April 11, 2018

My Five-Monitor MacBook Pro Setup

I currently have my primary work development computer, a 2017 MacBook Pro (15", with Touchbar), set up with a total of five monitors: The MacBook's own display, plus four external monitors.


Five monitors? Isn't that just a bit ridiculous?


Although it may seem like overkill, I have found that this setup is really good for productivity.  I'm able to put my most-frequently-used windows each on their own monitor, which is great for avoiding time wasted switching different windows back and forth into the foreground.

It's also great for helping to minimize the situation of making a mental note of something in one window, switching to a different foreground window, and then doing something in that other window with the remembered information; I can just glance back and forth between multiple visible windows instead.

My typical window configuration with this setup:

  • Second monitor from right: My primary monitor. Typically I've got a web browser here.
  • Second monitor from left (portrait / vertical orientation): My code window. Typically my IDE / code editor (e.g. Visual Studio or RubyMine) goes here. Great for seeing tons of code at once! Also excellent for reviewing long documents, such as XSDs or other documentation.
  • Leftmost monitor: Secondary browser or document window. I put terminal windows here. I also put my mini-mode iTunes music player in the corner.
  • Rightmost monitor: Communication / chat. I typically have my team's Slack window open here.
  • Top monitor: I keep a video conference window to my team's office in San Diego open here during the hours that they are working, so they can see and talk to me, and vice-versa!

What's the hardware?

Computer

MacBook Pro, 2017, 15".

As far as ports go, this computer has 4 USB-C ports, plus a 3.5mm headphone jack -- and that's it! Power, video, network -- everything is done through the USB-C ports.

I have the laptop sitting on a 3M LX500 Laptop Stand to align its screen a little better vertically with the other displays.  I like this stand because there's a nice empty space under the stand (which is the not the case with many stands), which is nice for a little extra physical storage space on the desk top.

USB-C to 17-port adapter

OWC 13 Port Thunderbolt 3 Dock.

This is the main device that allows me to connect so many devices to my MacBook, including two of the four external monitors, as well as to provide power to the MacBook, all through a single one of the MacBook's USB-C ports.

Primary external monitor (second from right) 

Dell U2312HM. 1920x1080 resolution.

Connected via: MacBook USB-C port → CableCreation brand USB-C to DVI adapter → DVI cable → Monitor DVI port


Portrait mode (vertical) monitor

Dell U2412M. 1920x1200 resolution. 

Connected via: OWC Thunderbolt Dock → BlueRigger brand Mini Displayport to DVI cable → Monitor DVI port  

The stand that comes with the monitor supports rotation, so it was very easy to get it set up in portrait mode.  

The bump to 1200 resolution in the narrow dimension (instead of 1080) is nice, as it gives the monitor a little extra horizontal room in portrait mode. 

Modern versions of MacOS and Windows both have excellent support for "rotating" the output for a given monitor to support portrait orientation -- I've successfully used this monitor in portrait orientation with both OS X and Windows.

Rightmost (tertiary, 4:3) monitor 

Dell E196FP. 1280x1024 resolution. 

Connected via: OWC Thunderbolt Dock Agatha brand USB-C to VGA Cable → Monitor VGA port


I wouldn't actually recommend this specific model of monitor anymore, because it only supports a VGA cable connection! Newer connection types such as DVI, HDMI, or DisplayPort will generally provide a sharper screen image. However, even with just the VGA connection, in practice I don't notice any bothersome blurriness on this display.

Top monitor: Dell S2415H. 1920x1080 resolution with integrated speakers.

Connected via: MacBook USB-C port Apple brand USB-C Digital AV Multiport Adapter → HDMI cable → Monitor HDMI port

Stand: WALI brand Extra Tall Universal Single LCD Monitor Fully Adjustable Desk Mount


How painful is docking / undocking?


It's not bad at all, actually; there are only 3 USB-C cables to plug or unplug when I connect or disconnect the MacBook laptop from the setup: The one USB-C cable for the OWC Thunderbolt Dock, plus the two additional monitor adapter dongles.

It's not quite as nice of an undock experience as when I used this same set of monitors and other devices with my Lenovo Windows laptop -- which involved unplugging just a single HDMI cable from the laptop itself, and removing the laptop from the physical dock -- but it's still very acceptable!


What's with all of the Dell brand monitors?


Experience has taught me that some non-Apple monitors just don't play nice with MacBooks. The typical symptom I've seen is that I'll connect a monitor to the MacBook, and simply nothing will happen: The Mac will act as though nothing new was connected. Even the System Preferences → Displays → (hold Option button) → Detect Displays feature wouldn't help.

I don't know of any reliable way to tell whether a particular non-Apple monitor is going to work with a Mac other than actually trying it.

I've had good luck with Dell monitors working with the MacBook, though, so my current plan is to continue going with the "Dell" strategy until something changes -- that is, until I run across a Dell monitor that doesn't work with the Mac!


So why the detailed write-up?


Actually getting this to work was the end result of a lot of trial and error.  I tried several times connecting these and other monitors to the MacBook in ways that resulted in things not working -- almost always manifesting itself as the MacBook acting as though one or more of the connected external monitors didn't actually exist at all, as I noted earlier.

Some specific things that I remember trying when I was getting this all set up that didn't work:
  • Trying to connect more than two external monitors via the OWC Thunderbolt dock. I was only ever able to get a maximum of two external monitors working through that dock; any further number of monitors I tried just wouldn't work.
  • Certain other adapter and cable brands. In particular, whereas the CableCreation brand USB-C to DVI adapter (which has only 3.5 stars on Amazon as of this writing) worked for me, a similar Belinda brand USB-C to DVI adapter did not work for me -- even though it specifically listed support for MacBook -- and I ended up returning it.
  • In general, I had a lot better luck getting "non-handshaking" port types (DVI, VGA) to work than "handshaking" ports (HDMI, DisplayPort). I had several different failures trying to get the two primary Dell monitors, both of which support both DisplayPort and DVI connections, to work with the MacBook via their DisplayPort connections, before I did get them working via their DVI ports. (Both Dell monitors did work just fine with my Lenovo Windows laptop via either one of their ports.)
Hopefully this post will prove of some value to anyone who might be trying to get a similar multi-monitor MacBook setup working; or to me, in the event that I need to try and recreate this setup in the future!


Wallpaper shout-outs


OS X does not currently have native support for spanning a single image across multiple monitors as the desktop wallpaper (like Windows does). The great $3 solution that I used to create the spanned image of the Enterprise-D as shown in the photo above is the Multi Monitor Wallpaper app, available on the MacOS App Store.

Multi Monitor Wallpaper shows you an outline of your combined monitor layout, and allows you to stretch / zoom / reposition your selected image across the monitors. It then automatically handles cutting up the image and setting it as the background image for each monitor. I have no affiliation with the app; I'm just a fan!

As for the excellent high-resolution image of the Enterprise-D itself, unfortunately I'm not sure whom to credit. I grabbed the image over a year ago via a Google image search, and I can't find the same image now when I search. If anyone knows where this image is from, let me know, and I'll gladly provide credit!

 



Tuesday, April 10, 2018

Fix: Blurry display on HDMI-connected Dell 2415H monitor on MacBook Pro

On the MacBook Pro (2017, with Touchbar) that I currently use as my primary development computer at work, after I "docked" the computer (reconnected all external monitors and devices) this morning, my monitors were in the wrong order.

Further, the display on one of the external monitors -- a Dell 2415H, connected via USB-C port → USB-C to HDMI adapter → HDMI cable → HDMI port -- had an extremely blurry display. That monitor also now had "overscan" of about a centimeter; that is, the image extended past the viewable area of the screen in all directions (so that I could only see the bottom edge of the menu bar at the top of the screen).

Fixing the display order was a simple matter of going into System Preferences → Displays → Arrangement, and dragging the monitors back into the proper order.

However, it took a little longer to figure out a solution to the blurriness issue. In the System Preferences → Displays window for the blurry monitor, the Resolution radio button was set to "Default for display" as usual. When I changed it to Scaled, the available values were "1080i" (selected), and 720p.

Changing the setting to 720p caused the display to be sharp again, but everything on that monitor ended up being sized way too large.  (Plus, I then was only presumably getting 720 pixels of vertical resolution, instead of the 1920x1080 of which the monitor was capable.)

The solution turned out to be to close the Displays windows, unplug and replug that monitor's HDMI cable, and then open System Preferences → Displays again.  After doing that, the Displays window → Color tab → Resolution (scaled) selection for the blurry monitor now displayed two additional options which weren't there before: "1080p" and "1600 x 900".



Changing the selection to the now-available "1080p" fixed the problem: All text and images on the monitor became nice and sharp again, and the overscan went away.

Monday, December 04, 2017

Advent of Code 2017 - SmartyStreets Sponsor Text

I just learned about the Advent of Code -- an event with a new short, fun programming puzzle published in each of the 24 days leading up to Christmas.

I noticed that primary event sponsor SmartyStreets has some cryptic text as their sponsor comment:

U2VuZGluZyBDaH Jpc3RtYXMgY2Fy ZHMgdG8gYmFkIG FkZHJlc3Nlcz8K

I won't spoil what it says here, but the only hint you should need to decode the text is that it is base64 encoded. (Also, automatic base64 decoder websites are readily available!)

Also, I'm publishing my C# Advent of Code 2017 solutions on GitHub. I'm generally aiming to optimize them for readability, rather than for minimum lines of code or fastest possible execution speed -- which is what I generally do as well when writing actual production code.  That being said, I have been pretty blown away by the impressive brevity of the solutions of some of the event leaderboard's top members!