The mystery of the broken JWT magic link login URLs on iPhone

My team at work was recently facing a problem where "magic link" login URLs being sent out via SMS (text message) were "broken" when received by iPhone users. Only part of the URL's query string portion was properly rendering as part of the link; the remainder -- despite not being separated by a space, or any URL-invalid characters -- was showing up as plain text:

A magic link in this context is an URL that includes a secure, tamper-evident key which identifies the user, and allows them to log in to the application that sent the link, in lieu of having to enter a password. (This has security pros and cons; that linked article provides a nice summary.)

My team is using JWT as the magic link key. JWTs are encoded into three portions, separated by period characters (remember that, it's important later!): A header; the message payload (including things like the user's ID, and the key's expiration time); and the signature verification.

In our case, only the first portion of the JWT value, the header, was being rendered by iPhone recipients of our SMS message as a part of the clickable hyperlink. The remaining two portions were showing up as plain text. This broke the magic link! While it still directed users to our site, it was unable to log them in.

I spent the day yesterday performing an investigation into why this was happening.

For starters, asking Google about the maximum length of an SMS message yields the answer "160 characters." Here in 2023, as far as regular users are concerned, this is no longer really true. (When's the last time you were composing a text message, and your phone stopped you from sending your message because it was longer than about half a Tweet?) All modern providers use "SMS concatenation" to, behind the scenes, break a long SMS message into multiple parts, and then seamlessly stitch those parts back together into a single message for the recipient.

I hypothesized: Perhaps Apple's implementation of SMS concatenation doesn't work when the URL itself is longer than 160 characters (as our magic link login URLs including a JWT token are)? No; I was able to disprove this by sending myself a text message with such an URL; it arrived in one piece, no problems. 

(As an aside, I started out doing these tests by using the web UI of my work's existing Twilio account to send messages to my personal iPhone's number. This worked fine; I pretty quickly determined, though, that I could more expeditiously test by just using my Mac's Messages app to send messages to my own phone number. This produced the same results, as far as the received message ending up broken or not.)

Perhaps SMS concatenation doesn't work when the query string portion of the URL is longer than 160 characters? No; disproved by sending myself such a link, which once again was delivered in one piece, as expected.

Perhaps the problem is when a single query string key-value pair -- or just a query string value -- is longer than 160 characters? No; I was able to successfully send myself messages (using the string "1234567890" repeatedly as the query string value to achieve the target length) in with such query string values excess of 500 characters in length, no problems.

My testing went on like this. I was consistently able to reproduce the broken link behavior using an actual (Dev environment) magic login link; the behavior of any particular URL being broken or not appeared to be consistent/deterministic, at least. Further, by trimming down certain portions of that URL, the link would be correctly delivered in one piece. 

By testing many message and URL variants, and recording for each one whether it succeeded or failed to deliver properly, along with the lengths of the various portions of the message text and the URL, I was finally able to pin down the problematic behavior. Here it is, in plain English:

For a given query string value: If that value contains any URL-valid punctuation characters (i.e. non-alphanumeric characters): If any portion (or "slice") of that query string value beyond the first portion, when separated/sliced by punctuation characters, is 302 characters or longer, the URL will break (on Apple devices). If all such portions are 301 characters or shorter, the URL will render correctly. 

Recall that JWT values consist of 3 portions -- separated by period characters? This meant that if the token's 2nd (payload) or 3rd (signature) encoded portions were in excess of 301 characters, the resulting link would be broken when delivered to an iPhone. 

(Notably: It's only Apple's handling of SMS messages, in their Messages / iMessage app on iPhone and on Mac, where links render as broken in this particular way. In my testing with Android clients, and with Google Voice, all links that I tested with were delivered correctly, regardless of length!)

Here are a few examples of working and broken URLs (when received by an Apple client). To save space (and to make this post less ugly!), instead of actually spelling out URL portions of 300+ characters, I'll represent such portions with the number of characters in that portion. The following links, when delivered to and viewed on an iPhone, or in Apple's Messages app on a Mac:

https://example.com?key=400 (OK; there are no punctuation characters in the query param value)

https://example.com?key=10.302 (BROKEN; the second portion of the query param value is longer than 301 characters)

https://example.com?key=301.301_301 (OK; no portion of the query param value is longer than 301 characters)

https://example.com?key=200~200.400 (BROKEN; the 2nd portion is ok, but the 3rd potion is longer than 301 characters)

https://example.com?key=400-50 (OK; only the first portion of the query param value is longer than 301 characters, and that doesn't manifest the problem)

https://example.com?key1=400&key2=400 (OK; the both query param values here consist only of "first portions", which don't manifest the problem)

To work around this problem -- and to produce links that are some what less nasty-looking on clients that render the entire URL -- I'm planning on making a pair of changes to our magic login tokens:

1. Reducing the payload content to "essential" values only. Namely, the user's email address, and an expiration date/time value. This will cut down on the middle "payload" portion of the JWT.

2. Using HS256 instead of RS256 as the signing algorithm. For our specific application and usage scenario, HS256 will provide sufficient security; but HS256 signature values are significantly shorter in length. 

All of the aforementioned testing was done in June 2023 using an iPhone 12 running iOS 16.5.1; and a MacBook Pro running MacOS Ventura 13.4.  Perhaps Apple will address this issue in future software versions? (But if this particular bug isn't at the top of their priority list, I certainly can understand why not. ☺)

Hopefully this post may be helpful to any of y'all out there who are researching why your SMS messages that include JWT magic link login URLs (or other long URLs including long query string values) being delivered to iPhone clients aren't rendering properly!

The Mystery of the NULL Values in a NOT NULL DATETIME MySQL Database Column

My development team at work yesterday picked up a task to investigate a error that was periodically showing up in our production logs:

NoMethodError: undefined method `>' for nil:NilClass

This is a common Ruby error that occurs when you try to call a method on a variable with a value of nil. In this case, the "method" in question was the greater-than operator, ">". 

The line of code associated with the error in question was actually a conditional evaluation that involved 3 separate ">" evaluations, which obscured the exact variable that was the source of the error a bit. 

Two of the three variables in question were attributes of an ActiveRecord model object whose associated database column was defined as NOT NULL. Those attributes were not being re-assigned after being read from the database, so we initially ruled them out as being the possible cause of the error.

However, further investigation revealed that none of the three variables on the problematic line of code could -- in theory -- possibly ever be null. Faced with this, I decided to take a closer look at the actual data in the database.

Our production environment is split into multiple MySQL databases. For purposes of conveniently being able to query customer data across databases all at once, we have an ETL process which extracts (non-sensitive) customer data from the source databases, and populates it all into a single central Snowflake SQL database. 

The database structure of the relevant table (Users) was similar to this (simplified for brevity):

• id: INT, primary key
• first_name: VARCHAR(255)
• last_name: VARCHAR(255)
• updated_at: DATETIME, NOT NULL

Temporarily putting common sense aside, I queried the snowflake database to see if any of the NOT NULL updated_at values were nevertheless null:

SELECT * FROM Users WHERE updated_at IS null

This returned 0 results, as expected. 

Acting on a hunch, I tried searching for unexpectedly old records; this database has been in service since about 2009:

SELECT * FROM Users WHERE updated_at < '2005-01-01'

This query did produce some results! Out of the tens of thousands of records in the Users table, a few hundred records were returned whose updated_at date was '1970-01-01 00:00' -- a value equal to the start of the epoch in Unix time. 

At this point I questioned: Does Ruby on Rails and/or ActiveRecord somehow treat start-of-epoch date values of '1970-01-01 00:00' at nil? This seemed unlikely, but I tested it anyway, setting the updated_at value for an existing record in my local machine's copy of the database to that start-of-epoch value; and then reading the record into the corresponding ActiveRecord model object. Not unexpectedly, the updated_at attribute on the model did not end up with a nil value; it had the expected value of midnight on January 1, 1970.

I still felt like I might be onto something here, though. At this point, I wanted to inspect the data in the actual MySQL customer database. I don't have access to the live database, but I was able to access a read-only replica of the database.

Having connected to that MySQL database, I verified that the updated_at column was still defined at NOT NULL; it was. I then ran the same query to look for old values:

SELECT * FROM Users WHERE updated_at < '2005-01-01'

As before, this returned a few records -- about 300. However, I noticed that the actual updated_at values were different. This time, they were returned as '0000-00-00 00:00'. 

Year "0", month "0", day "0" -- all nonexistent values. Feeling a bit of a chill, I re-ran my earlier query to look for records with null values:

SELECT * FROM Users WHERE updated_at IS null

MySQL returned the same 300 records. 

Cool, cool, coolcoolcool. The evidence indicated that (1) MySQL was allowing values of '0000-00-00 00:00' to be set on a DATETIME, NOT NULL column; and (2) despite the aforementioned NOT NULL restriction on the column, was evaluating such values as being null.

To close the loop, back on my local development machine's database, I set an existing record to the '0000-00-00 00:00' value -- which MySQL happily allowed -- and then, in the Rails console, populated a model object from the record. Sure enough: The updated_at attribute on the record had been assigned a value of nil.

Summary of findings

(1) In a Rails application (and possibly in other languages / frameworks as well), it may not be safe to assume that a value read from a DATETIME, NOT NULL column on a MySQL database is actually guaranteed not to be null. MySQL allows a value of '0000-00-00 00:00' to be set in such columns (despite the NOT NULL restriction); such values are treated as null both my MySQL itself, and by Rails / ActiveRecord.

(2) Although I'm not familiar with its inner workings, that the ETL process from the multiple production MySQL databases to consolidate data into a single Snowflake database couldn't 100.0% be trusted not to change data values. In this case, it silently converted MySQL DATETIME values of '0000-00-00 00:00' to values of '1970-01-01 00:00' 

Mitigations

Doing a bit of additional reading after the fact, I found a StackOverflow post that references a MySQL configuration mode of "NO_ZERO_DATE," which when set, prevents such "zero date values" from being set. 

According to the MySQL documentation, not setting that option may be "more convenient" and/or consume less space. Based on my team's experience here, though, I'd certainly be inclined to take advantage of that NO_ZERO_DATE setting whenever feasible!

The Jon Schneider Game of the Year awards: 2022

Starting with my personal Game of the Year for 2022, and continuing on with the rest of the top 10, these were my favorite games that I played for the first time this year (along with the platform(s) on which I played them)!

1. Persona 5 Royal (Switch)

Probably the most expansive single-player RPG I've ever played, I was hesitant to dive into the this game given the expected time commitment, but I'm certainly glad I did! Engaging turn-based combat system, check. Interesting story, check. Realistic and sympathetic characters, check. Getting to explore (in a limited way) the districts of real-life Tokyo, a fun bonus. All wrapped up in a package that tries hard to be super stylish and cool -- and actually manages to pull it off!

2. Xenoblade Chronicles 3 (Switch) 

I enjoyed seeing the lore of the first two main games in the Xenoblade series woven together here into something completely different, yet still congruent with the original pairs of source material. A key innovation here was an RPG with a large party of player characters where some PCs aren't present during combat; all 6 (or 7) of them are present, contributing, and playable. The way the game handles, in consistent fashion with the game's world, a mixed-gender party of (essentially) late-teens / early-20s characters with romance between characters (mostly) not being a thing was also an enjoyable novelty. 

3. Crystal Project (Steam / Mac / Windows)

This game was a real treat for me, a fan of old-school Final Fantasy games and similar JRPGs. Layering challenging turn-based on top of exploration of a blocky Minecraft-like world worked remarkably well. Like Stardew Valley a few years back, Crystal Project is a game that managed to pleasingly surprise me on a number of occasions after I thought that I had what the game had to offer all figured out.

4. Knotwords (iPhone)

The best word game app I've ever played. It's Sudoku crossed with crossword puzzles in a really smart way. Wonderful UX design (which inspired a Vigil RPG haptics update!). More-than-fair premium pricing model with no ads, premium currencies, or other typical mobile-game dark patterns. Multiple game modes, including daily and monthly puzzles. Knotwords' ability for the player to "recover" lapsed daily streaks is another innovation that should be universally copied. Highly recommended to fans of word games!

5. Isle of Arrows (iPhone) 

Another premium iOS game done right. Combines tower defense, a light puzzle/card game, and a cool aesthetic. I 100%'ed the campaign.

6. Phoenotopia Awakening (Steam / Mac) 

A smartly-designed, difficult-but-fair, cute pixel-art hybrid of Metroidvania and Zelda 2: The Adventure of Link gameplay styles, with great settings/locations and story.

7. Kirby Forgotten Lands (Switch) 

I played through this cute first 3D foray of Kirby through start-to-finish with my 10-year-old, with her taking on the role of Kirby, and me, Bandana Waddle Dee. I appreciate how this game caters to a variety of skill levels, with the the main story being well suited for the pair of us to complete, and also including a post-game suitable for experts.

8. Chasm (Steam / Mac)

Like Phoenotopia, a pixel art melee Metroidvania game, whose weapon/combat system seems to have been particularly inspired by the more recent 2D games in the Castlevania series (see: next entry below!). Slightly thin on story, but solid enough in terms of level design and gameplay that I did complete the main story.

9. Castlevania Aria of Sorrow (Switch port of Game Boy Advance game) 

Played through this game which I'd never played in its original form as part of the Castlevania Advance Collection. I'd previously been most familiar with the more traditional series entries including the original Castlevania (NES) and Super Castlevania (SNES); hat tip to TouchArcade and freelance game writer Shaun Musgrave, whose recent Patreon ranking of all(!) of the Castlevania series games inspired me to give this a play-through. 

10. Super Auto Pets (Web / iPhone)

This is the type of multiplayer-game that I tend to prefer these days: Quick and asynchronous, meaning low-stress! You start each game from scratch to build a team of up to 5 of animals, each with their own unique combat ability, and watch them engage in a series of brief auto-battles against teams built by other players who have done the same.

Workaround: Input Lag with Nintendo Pro Controller playing Steam games on Mac

I was having a problem where, when playing the game Phoenotopia Awakening on my MacBook using a Nintendo Switch Pro Controller connected to the Mac via Bluetooth, I was getting pretty significant "input lag": About half a second would elapse between me pressing a button on the controller, and the character reacting on the screen, making the game nigh-unplayable.

Oddly, this issue would only occur with my MacBook "docked" -- connected on my desk to my multiple monitors, wired ethernet, and various other peripherals. When using the MacBook as a standalone laptop, the game and controller would play just fine.

Even in the "docked" configuration, however, outside of Steam, an in-browser gamepad tester showed that the controller was communicating with the MacBook with no lag.

I found the following to be an effective workaround for the problem:

1. In Steam (not in "Big Picture" mode, and with the game not running), locate the game in your Library.
2. Right-click the game and select Properties.
3. On the Controller tab, select "Disable Steam Input".

 Voila -- No more input lag with the Nintendo Switch Pro Controller, even when docked! 

 Hat tip to "Toms33" for posting this solution in a thread on the Steam Community forums. 

TimeSnapper: The power of being able to replay the past

To get this disclaimer out of the way right out front: I'm the author of TimeSnapper for MacOS. But like the guy in the 1980-something Hair Club for Men TV commercial: I'm not just the author, I'm also a client! A real event just happened where TimeSnapper enabled me to share a key bit of information with my team at work that I hadn't been able to come up with otherwise.

About a month ago, I came across a mention from a co-worker of a book as being a great introduction to the FinTech space: "The Anatomy of the Swipe" (2020) by Admed Siddiqui. At the time, I checked my local public library for a copy, but they didn't have one; so I took advantage of an awesome feature of the library, and put in a request via the library's website for the library to purchase a physical copy! They did so, and automatically put the book on hold for me! I picked up the book yesterday.

In a team meeting near the end of my work day yesterday, I mentioned the book, and promised to drop a link to the book in our Slack channel in the morning. I followed up on that this morning, intending to post the link along with the original post where I'd seen the book recommended internally.

I searched Slack for the earlier mention of the book that I remembered, and... nothing. 

No search results for "anatomy of the swipe". Or for "anatomy" "swipe". Or for "ahmed siddiqui".

Although I thought I'd seen the book recommendation via Slack, maybe it had actually been via shared Google Doc...? Or an email? I searched those places too, and... again, nothing.

At this point, I was momentarily stumped. I wanted to provide credit where it was due and share the original book recommendation I had followed... but where was it? Obviously I hadn't just imagined it; I had gotten the name and author of this particular book, in order to request my local library to order a copy, from somewhere.

And then, inspiration struck: I've got TimeSnapper! TimeSnapper, if you're not familiar with it, is a utility program that runs in the background on your computer, and saves a snapshot of the screen every few seconds. It also provides a "Play Your Day Like a Movie" feature, where you can rapidly scrub back and forth through an entire day's worth of these saved screenshots.

To get the time window for my TimeSnapper search, I went to my library's website, logged in, and and viewed the page with the status of my past purchase requests. Sure enough, it showed that my request for "The Anatomy of the Swipe" had been originally made on July 26, 2022 -- one month ago today.

I then fired up TimeSnapper's Play Your Day feature and navigated to July 26. Skimming through my day, I found the images of my visit to the library website where I'd initially keyed in my request for the book. Navigating a short way back in time from there, I saw myself visiting my company's Slack instance... and found the recommendation I had remembered -- not for the book itself, but a link to a blog article with a list of good FinTech introductory resources -- which is where the mention of the book has actually been located!

Thanks to TimeSnapper, I was able to post a link that Slack post with the link to the FinTech blog post along with the link to the book itself.

Obligatory closing promotional line: If this ability to go back in time and observe what you were seeing on your computer in the past sounds like a superpower you'd like to gain for yourself, you can read more about TimeSnapper, and buy a copy for Mac or for Windows, on the TimeSnapper website!