šØāš» Steel Orca Tech Support #1: I broke Substack (and it wasn't that hard)
š How a technology company's production support operates is a key indicator of the health of their overall business. Last week suggests Substack's model is broken. It begs us to ask, "What else is?"

Welcome to the inaugural issue of Steel Orca Tech Support. In this series, we document the technical challenges we encounter in our day-to-day dealings with the internet and technology. We are not claiming expertise, only seeking knowledge and wisdom. If youāre reading this and have the answers we donāt, leave a comment below.

In my Songs of the Week #13 post, I stated the following:
I had some technical issues with Chrome after Substack made the latest update, but I didnāt get anywhere with Substackās ticketing system or Twitter, so I switched to Edge. A bit disappointing, but the platform is free (beyond my paying for the domain, which was a one-time deal). Iām getting what I pay for, and other people were at least not reporting the problem, so maybe itās all my fault.
That ended up being somewhat prescient, as upon posting on Friday, November 16 at 10 am MT, that post broke both the iOS app and the Android app. At the time of writing, Songs of the Week #13 still will not render on the Android app.
To direct my energy toward something positive, I thought it would benefit everyone if I documented my experience with Substack technical support. The two resources Iām aware of for end users are the Substack Status page (to monitor incidents) and the Substack Help Center, which is powered by Zendesk.
In addition, while Iām not a DevOps or IT support person by profession, I do āknow enough to be dangerous,ā as every middle-manager IT hack says multiple times per week. So I have also applied severity (S1-4) and priority (P1-4) as rough estimates.
Finally, these posts will be living documents, meaning, if an incident is still ongoing at the time of publishing, I will still publish, and the post will just be updated (without re-sending to everyone) based on subsequent updates that I receive or gather through additional testing (to the extent I do any). I hope you enjoy and find this helpful, and I will try to keep it positive.
The Steel Orca is a reader-supported publication. If you enjoy this post, consider becoming a free or paid subscriber.
Essential Context: āSubstack is powered by Ghost.ā
On December 12th, John OāNolan, Founder/CEO of Ghost (one of Substackās main competitors), posted the following Twitter thread:
Some personal news: Iām thrilled (and admittedly, somewhat surprised) to announce that as of Friday, @substack is now officially powered by @Ghost!
ā John O'Nolan š“āā ļø (@JohnONolan) 3:51 PM ā Dec 12, 2022
Keep reading for an exclusive behind the scenes look at the engineering around the launch of THE TWITTER FILES PT2: š§µ
I am not going to go through it and explain the whole thing. But, essentially, it looks like the Substack development team got caught red-handed, effectively copying large amounts of Ghostās code (and not attributing it correctly) to make a big splash with the release of Bari Weissā new publication, The Free Press.
At the end of the thread, OāNolan calls out Substackās co-founder Chris Best. Chris replied in a thread of his own:
A response to @JohnONolan here to clear up some serious misunderstandings
ā Chris Best (@cjgbest) 6:30 PM ā Dec 12, 2022
https://t.co/biU4LQvass
In my opinion, Chrisās response was less than convincing. This tweet from him near the end gives you the gist:
Weāre grateful to the developer of the Tripoli theme and to Ghost for its contributors to open source work. Weāre exploring ways to give writers more customization on Substack. This is one approach weāre considering but itās too early to know if weāll scale it up.
ā Chris Best (@cjgbest) 6:30 PM ā Dec 12, 2022
To which this random dude cheekily replies:
@cjgbest "No attribution and non-compliance with licensing terms is the sincerest form of flattery" - this dude
ā N628TS on ADB exchange (@chrismerkel) 4:21 AM ā Dec 13, 2022
Again, this is essential context based on when the issues below started. I understand they may be and likely are separate. Still, from what I can tell, the Substack development team is under intense pressure to deliver new platform features as quickly as possible. This is especially true in light of the additional context that Twitter has indicated they will shut down their newsletter service Revue on January 18, 2023. If I were a Substack executive, Iād think itās time to strike while the ironās hot.
Current Bugs
One note on the bugs below - from what I could tell, the Substack Help Center does not generate any ticket number or request ID for the customer upon submission or when Zendesk generates the automated support email. This doesnāt seem great. The only way I got the ticket numbers below was by going into the email HTML and getting the support ID thatās auto-generated for the āReply Toā email (see below):

Thatās not something anyone will do, so upon submission, you cannot track a follow-up on your help desk ticket. Send one and pray. Hopefully, your issue isnāt too serious. Not a great look.
S3P2 - āDuplicate to draftsā freezes Substack editor in Google Chrome (Current Status = Open)
Ticket# 231591 - Support Request Received Dec 18, 22:38 MT
The first thing to mention is that this was the original bug I encountered last week, but I wanted to do my due diligence for the Substack team and create and document a support ticket before making this post. So the tickets were submitted out of order, but chronologically this is how it went down.
As mentioned above, Substack has been releasing a lot of new features. One of which was āDuplicate to draftsā:
Product news:
ā Substack (@SubstackInc) 8:58 PM ā Dec 12, 2022
Writers can now "duplicate" posts, which enables you to easily re-use templates instead of reformatting each post from scratch.
Just click "..." by one of your posts and "Duplicate to drafts."
That tweet excited me as my Songs of the Week series has a standardized format I manually recreate each week. This would save me a bunch of time.
Except when I went to click on it and use the functionality, it didnāt work. Immediately my browser froze. I tried it again, thinking it was an issue with my machineās performance, and still nothing. So I tried it in another Chrome session, and it froze immediately.
Whenever I tried to duplicate a post, it would reproduce successfully as an object on my writerās dashboard, but it would be utterly unopenable in Chrome (even to delete). After complaining to Substack on Twitter about this (and the āSubscribe with Captionā button issue described below), I then figured out a workaround where if I opened the same duplicated post in Google Chromeās mobile browser on my phone, I could at least get to the section where I could delete the posts. So I did that, given that I didnāt want a bunch of garbage drafts sitting on my Substack writerās dashboard.
After doing that, Substack replied on Twitter:
@steelorcaio @Ghost @hamishmckenzie Thank you for letting us know! It looks like you were able to delete the extra duplicates. We're making some changes to button, so that these sorts of snafus don't crop up again.
ā Substack (@SubstackInc) 11:18 PM ā Dec 13, 2022
At that point, I just left it as I figured, āSubstack is free, donāt worry too much about it; youāve got a workaround.ā
S3P2 - āSubscribe with Captionā button freezes Substack editor in Google Chrome (Current Status = Open)
Ticket# 229777 - Support Request Received December 14, 15:45 MT
When I discovered the āDuplicate to draftsā issue above, I figured, āWell, who cares? I didnāt have this functionality before today, anyways. Iāll do what I planned and recreate the post manually.ā
To drive subscriptions (as one does on Substack), I include a āSubscribe with Captionā button near the top of each newsletter. No big deal. Why donāt you try the one below to see how easy it is if you havenāt used one before?
The Steel Orca is a reader-supported publication. If you like this post, please consider becoming a free or paid subscriber.
Unfortunately, my editor session completely froze when I clicked on the āInsert subscribe button with captionā on the Editor toolbar. Iāve just tried it in Chrome and am still having the same issue. However, everything seems to work just fine in Microsoft Edge.
S2P2 - Post Does Not Render on Android Substack App (Current Status = Open)
Ticket# 230613 - Support Request Received December 16 10:48 MT
⦠and this is where the proverbial sh*t hit the fan, as it were. Because of the above technical issues, I had to work on Fridayās post until around midnight on Thursday, then went to bed and worked on it another couple of hours until just before posting time. Having switched to Edge, everything seemed fine. I sent myself a few draft posts, and it seemed all good. Ok, great.
I completed my final reviews of the post at around 9:45 am MT, scheduled it to send out at 10 am MT (9 am PT, 12 pm ET) to keep it consistent, and then went and did some other work.
I get the email to both my Android and iPhone, and everything seems fine. The email looks excellent, and everything seems to work.
At around 10:15 am MT, Clara Beth (my publishing partner in charge of photography) then messaged me saying the iPhone post was rendering weird and that I should check it out. So hereās what I got:

I then tried the same post on the Android app, and it just caused the Substack app to quit altogether. Unfortunately, Android doesnāt surface the error details, but the assumption is that itās a similar issue.
No one is going to paint me as some IT guru, but hereās what all thatās telling us - Substack is not returning valid JSON for a post that was able to be successfully created and published via their in-browser editor, which is one of their primary value drivers.
At this point, I started blasting Substack on Twitter because Friday is the only day I publish, and this was going to kill my view count (spoiler alert: it did). So I DMād Substack and posted a few times but didnāt see any reply (and they were continuing to retweet advertisements for random new accounts during this time). So not having any luck, I decided to go straight to the top:
Hey, @hamishmckenzie any idea on this?
ā steelorca.io (@steelorcaio) 6:05 PM ā Dec 16, 2022
Maybe you guys imported too much of the @Ghost code without checking it?
Is there someone else I should be reaching out to? I get that the application is "free" but your IT support doesn't respond and just makes me feel like I'm crazy. š¤Ø
Now Hamish is a good guy and was a total gentleman about the whole thing. When I posted the actual JSON error above, he reacted quickly:
@steelorcaio @Ghost I'll look into this.
ā Hamish McKenzie (@hamishmckenzie) 6:32 PM ā Dec 16, 2022
I donāt really like Twitter, to be honest, but as you can see above, this was the quickest way to get someone of some import at Substack actually to look into the issue. And my line above about āMaybe you guys imported too much of the Ghost code without checking it?ā is not a nice thing to say (especially for a Canadian), but itās lines like these that get eyeballs on Twitter, and thus, reactions from the bigwigs.
@steelorcaio @Ghost quick response in the meantime: it seems to be an emoji-related bug. If you strip the emojis out, the problem should desist
ā Hamish McKenzie (@hamishmckenzie) 6:45 PM ā Dec 16, 2022
So Hamish talked to his development team, and they seemed to indicate an issue with emojis rendering on their app. That suggests that the problem was just introduced during the last major update (which happened around December 10, from what I can tell), as including emojis in all of my posts is standard, and none of them have had issues except this last post. Songs of the Week #13 isnāt even my most emoji-heavy post, so something substantially changed in the code so that emojis cause it to not render. The weird thing is my old posts with emojis still work fine.
I would strip out the emojis as Hamish recommended, but Iām not sure thatās truly the issue, and thatās fundamentally asking me to make a design change to my newsletter to accommodate undocumented tech limitations on his newsletter platform. That doesnāt seem right to me. Furthermore, if I change the post, I will have no idea when an update has been pushed that resolves the issue, which I can then update everyone here on. This bug is a secret portal to the inner workings of Substackās Android application development team.
I continued to test the post on both iPhone and Android throughout the day because, even though this is a free āhobbyā newsletter, Iām trying to do something here, and this stressed me out. Finally, about two hours later, the error was resolved for the iPhone, and that started to work, so half the problem was resolved. At the time of writing, however, this post still doesnāt render on Android. Iām assuming thatās because Swift allows quicker updates than Android, but Iām not an expert. If you know, leave a comment below.
One last update from this weekend - Substack Support replied to this ticket at around 5 pm on Saturday, December 17, with the following, which seems to indicate there has been a surge in support requests recently (emphasis mine). It looks like they chose their words carefully with the word contact:
Hi there,
Substack Support here. Thanks so much for reaching out, and apologies for the delay in response! We've had a higher volume of contact than usual and have been working to get to your message as quickly as possible.
I've reported this issue to our engineers so they can look into this further. Hopefully, they'll have an update soon.
We appreciate your patience and apologize for the inconvenience.
Best,
Substack Support
Implications
Misleading Incident Reporting
At the time of writing, the Substack Status page, which, again, is seemingly the only customer-facing page providing such information, has the following detailed as āPast Incidents,ā which also apparently includes current or ongoing incidents:

The fact that no incident has been listed when I have confirmation from the Substack co-founder Hamish McKenzie himself acknowledging thereās an issue (as well as the potential cause) on December 16, as well as a similar confirmation from Substack Support on December 17, indicates a fundamental disconnect in the management of incident reporting at Substack, or what information they are choosing to provide to their customer base at scale. Iāll let you be the one to decide which one it seems to be (if not both).
Editorās Note - Before publishing, an incident for December 19 related to the editor was logged, so weāre moving in the right direction -

I checked, however, and this fix didnāt address any of my issues. I would also call out that itās not clear from the incidents what the problem is, so this reporting is useless for the end user (especially if youāre relying on Substack for income).
Questionable Code Reviews
One of the first things people will tell you when you ask them about writing on Substack is the ease of editing and publishing. I am writing this on Substackās editor right now. I love it and would prefer to stick with this moving forward.
The issue is that if they are breaking their editor and their posting platform while simultaneously copying code from direct competitors without proper licensing or attribution, the development team either isnāt that good or is under tremendous pressure to deliver. Who is in charge that would allow that to happen? Who is approving the pull requests? What is the review process?
In particular, in my opinion, Chris Best and Substack need to reply specifically to the tweet below. I looked today, and they have not (I assume for legal reasons, but I am not a lawyer.)
One thing thatās a little disappointing:
ā John O'Nolan š“āā ļø (@JohnONolan) 3:51 PM ā Dec 12, 2022
Ghost uses the MIT license, one of the most permissive OSS licenses there is. Essentially, anyone can do anything they want with our code, with ONE basic requirement: You must include copyright attribution.
Which they have not.
Substackās Ability to Scale
As noted above, because Twitter announced they were shutting down Revue in early 2023, this represents a significant opportunity for Substack to become the market leader in email newsletter platforms. They are leaning into this from a marketing standpoint, but the above indicates that they are stretching their engineering and architecture teams. Thatās understandable, but that shouldnāt mean introducing platform-breaking bugs and questionable code.
Final Thoughts
I like Substack. At this point, Iāve paid for a custom domain and have all of my posts (all 14 of them!) on here, hopefully generating traffic. I donāt want to leave.
However, this past week, as well as some other technical challenges that Iāll detail in further Steel Orca Tech Support posts, indicate that while Substack might be the most popular right now, it may not be the best run from a development perspective. Hereās hoping they can turn that around sooner than later.
Even though the focus at Substack is on high-quality writing, podcasting, and video publishing, itās easy to think that Substack isnāt fundamentally a tech company. But thatās precisely what it is, and if it doesnāt start acting like it, it will lose its market share as quickly as it gains it, and people will start looking at Ghost more intensely.

Comments ()