Ever since I suffocated my dreams of being an indie game developer about a year ago and left (was chased out) to pursue other interests (ran like a yellow bellied coward) I've been dealing mostly with data analysis / content discovery. In fact, I recently stealth-launched the first site to reach a somewhat functional state, Let's Play Nexus. It works and I use it myself on a daily basis to find new videos, it's sort of awesome to realize you've designed something not just for others but yourself as well. Which is surprisingly uncommon for programmers I suspect.
LPN isn't done, in fact it's lacking several key features, which is why I haven't promoted it at all. But I've sent it to some Let's Players that I enjoy and received a ton of positive feedback. In general it appears people like sites that help other people find out about their own work, who would have known?
I found this fact very encouraging, especially since I'm working on half a dozen other similar sites but for other topics. Now, parsing big data from YouTube has been.. challenging. And I haven't really started yet. The sheer quantity of data that Google throws at you is staggering, making sense of it daunting. But the major issue hasn't at all been related to the process of developing the site, its back-end or even cobbling together the lifeboat to prevent myself from drowning in all the data. No, it's been choosing how to implement it.
C was my language of choice for the longest time. But as the web grew the mass of protocols and formats grew with it. Learning each one and implementing it in c was.. a lot of work. At some point I gave up and decided to drink the Kool-aid. I switched to PHP. And I hated it. But I hated it less than the alternatives and despite some major hassles with certain builds / configurations by certain providers the whole thing generally worked.
It's been many years since I ran a site beyond my blog, in fact my blog was down for many years (ahem). During that time I found it very relaxing to not have to worry about vulnerabilities cropping up in libraries, extensions and scripts. In fact, I've intentionally decided to not rely upon third party server-side tech or scripts this time around. My blog is powered by Pelican and runs locally, outputs flat html which my local server then synchronizes to this host. Works great. In fact, it works so great that I wrote my backend for Let's Play Nexus in Python and designed it for local use.
But I don't want to go back to the problems of yesteryear. I really, really don't want to venture back into the snake-pit that is PHP (and please, no, asp containers with vb, c#, et alles is just as unappealing). What do I do then? I've experimented with Python and it's fantastic for small web applications (and almost every type of non-web application, of course). But using it to build a CMS-like environment? Man, that was painful. node.js is very attractive, but it's really designed for real-time applications and offers no real benefits without the previously mentioned drawbacks.
Services like disqus help you add dynamic aspects to sites. Of course, the observant reader will note that it opens up an entirely other fiendish trap since you're basically giving a third party your data and hoping they don't go out of business / terminate your account. But with some care you can maintain local backups of all that data. Good grief this entire thing is rubbing me the wrong way.
I suspect this particular subject will crop up regularly on my blog if I continue doing web development. A solution has to be devised but what form it will take is currently shrouded in a haze of alternatives, each worse than the next. And to think I started out with the noblest of intentions.