How I failed to use the Twitter API to build a personal project

Well, it’s been almost a year since I wrote something here. The truth is, this garden has been abandoned. But the seed is still there.

Today I’m going to talk about how I failed to use the Twitter API to build a personal project. Or maybe I’ll let Copilot autocomplete my sentences and see what happens. I just recently added it to VSCode and it’s the first time I’m writing markdown with it enabled. It’s amusing.

I have a real problem with the Twitter API. I'm not sure if it's a problem with the API itself or with my code. I'm not sure.

Ok, no. What I want to say is that I have a real problem with hoarding tabs and links that I ‘cannot’ let go of. My reasons for that usually fall into two categories:

I have consumed the content and would like to keep it for future reference (articles, tweets, etc)
I would like to go back to this with time and study it (docs, youtube channels, services that might be interesting, etc)

I’ve tried keeping bookmarks, lists, databases in Notion, you name it. But what ends up happening is ‘out of sight, out of mind’. I forget about them and just start hoarding new ones.

My most recent struggle with this are Twitter threads. There’s a lot of interesting content people share in threads, but it’s the hardest type of content for me to keep track of. You need to go and look at each reply to get the full thing and there’s no easy way (that I know of) to scan the content and save it for future reference other than keeping that tab open forever.

So yesterday I had the idea of trying to use the Twitter API to build something that could transform a thread into one page of content (that I could maybe then save here in this website). My thought was to create a simple application with an input field for the link to the original tweet and then fetch the replies and render their content. ‘Simple’. Famous last words.

I had never used the Twitter API for anything, so I started by the docs, which were pretty easy to follow. Sign up with your existing Twitter account, get the API keys and start making requests.

It seems like the current API version (v2) is quite recent, so there was a lot of explanation in the docs about ‘the new way things work’ and that was a bit confusing sometimes as someone who doesn’t know how v1 used to work (turns out that’s probably how I wanted it to work).

In any case, there are three access levels: Essential, Elevated and Academic. I didn’t think what I wanted to do required too much as I really just wanted to read data from a given tweet so I signed up for the Essential level. I used Insomnia to make the API calls and it was pretty easy to retrieve the content and metadata for a specific tweet based on the tweet ID.

GET https://api.twitter.com/2/tweets/1479328649820000256

{
	"data": {
		"possibly_sensitive": false,
		"created_at": "2022-01-07T05:46:49.000Z",
		"lang": "en",
		"id": "1479328649820000256",
		"source": "Twitter Web App",
		"text": "Life is to short to use dated cli tools that suck\n\nTry these instead 🧵",
		"reply_settings": "everyone",
		"public_metrics": {
			"retweet_count": 4025,
			"reply_count": 210,
			"like_count": 17871,
			"quote_count": 372
		},
		"author_id": "2970738590",
		"conversation_id": "1479328649820000256"
	},
	"includes": {
		"users": [
			{
				"id": "2970738590",
				"name": "Amila Welihinda",
				"username": "amilajack"
			}
		]
	}
}

But the metadata didn’t have any information about replies. I searched the documentation to find out about conversation_id, which is a field added to the metadata of each reply to link it to the original tweet. I was then able to retrieve the original tweet from a reply, but still no way of getting the replies from the original tweet!

GET https://api.twitter.com/2/tweets/1479328652244373507?expansions=referenced_tweets.id

{
	"data": {
		"id": "1479328652244373507",
		"referenced_tweets": [
			{
				"type": "replied_to",
				"id": "1479328649820000256"
			}
		],
		"text": "`bat` is `cat` with syntax highlighting and line numbers\n\nhttps://t.co/q6ubwN86cE https://t.co/7bVASHjpbv"
	},
	"includes": {
		"tweets": [
			{
				"id": "1479328649820000256",
				"text": "Life is to short to use dated cli tools that suck\n\nTry these instead 🧵"
			}
		]
	}
}

The documentation also used the /search endpoint to retrieve conversations. That’s only available for the Elevated access level so I went to the dashboard to request the next level which was granted immediatelly after submitting the form.

I happily went back to Insomnia, using the Twitter API request builder to generate the requests I wanted to make. It still didn’t work.

GET https://api.twitter.com/2/tweets/search/recent?query=conversation_id%3A1479328649820000256

{
	"data": [
		{
			"id": "1483464532085329921",
			"text": "@amilajack Worth noting\n\n- gqh for hassle free Git cloning, https://t.co/Xzvn2GZhMD\n- peco is interactve grep https://t.co/R9N4APFnr1"
		},
		{
			"id": "1483130580845932547",
			"text": "@amilajack @tayhimself Thank you for this thread 😀"
		}
	],
	"meta": {
		"newest_id": "1483464532085329921",
		"oldest_id": "1483130580845932547",
		"result_count": 2
	}
}

The Elevated access actually only grants search access to the recent history, as in the last 7 days. If you’ve ever been on Twitter, you know things move way too fast for this to be helpful. The thread I was trying to get wasn’t even that old (beginning of the month), but it was already impossible to rely on whatever I could fetch from the last 7 days for this to work.

Back to the docs, historic search is only available for Academic access. Unfortunately you need actual academic reasons to get it.

Up until this point I had tried to keep to the docs and discover things by myself, so I finally gave in and searched for existing solutions. And now I know there are a couple of services that do that. I actually remembered seeing people using them on Twitter.

One of them (ThreadReader) even does the exact same thing I wanted to do (paste a link and generate a page), and that makes me think this was possible with the v1 API. But thinking about it now, these services mostly rely on the user replying to the thread and tagging their bot, which can then be used to track back all the previous replies. This approach could be worth looking into as well eventually.

It certainly frustrated me that I couldn’t get anywhere with my project this weekend because more than just being able to use it, I wanted to build this myself for learning purposes. But although I didn’t achieve what I intended, this experience motivated me to write and writing made me realize how much I actually did learn in the end!

Written on a Sunday in January, 2022