I just converted user data to SQL. This is a big change, please watch for bugs, and make a post tagged #bugski (or a github issue) if you see any.

This was the last big SQL migration. There are a few small things not in SQL, and a number of fields that need de-duplicating. But this was the last big destructive change.

If you've been following #devski, my plan hasn't really changed, but to reiterate, with this last big SQL migration done (assuming nothing explodes), everything else can be done in small pieces. Which are, roughly:

1. API logins. With user data and password hashes in SQL, implementing API logins should be simple. Which let us serve private data like mail, and your personal feed.

2. API pages. I prototyped this some months ago. This mostly means making SQL views to get all the data for a particular page, and the API endpoint to serve it. The view is the hard part. Theoretically, if we make views and endpoints for every page, we can make a static site of pure HTML and Javascript that simply queries the API. Or you can. Or a mobile app.

3. Data de-duplication. There's an absurd amount of duplicate data, everything from badges, to 'user_sharing' and 'user_shared_by', to 'muting' and 'muted_by'. Racket/SQL functions need added to e.g. get the posts badged by a user, and the users who badged a post, from a single "badges" table. And the duplicate fields and tables deleted. As most of you know, duplicate data is the bane of my devski existence.

4. Converting Arc to Racket. With the last big file loading functions in Racket/SQL now, it's much easier to convert Arc functions to Racket one at a time, from the bottom up. We could skip this entirely, if we decide to fill out the API and make a static site. Or I may do them anyway, at the same time. Because they bother me.

5. Get rid of the last files still used. Cookies, tags, probably a couple more. So that all Hubski data is in a single SQL database. This helps the other tasks listed, as well as making our lives easier, making backups easier, making it easier to write a completely new app from scratch. It makes a huge number of things easier if everything is in a single database.

6. Synchronise github. With all the Arc/app work I've been doing, I've written a lot of Racket that isn't in the github repo yet, just because I haven't had time. My plan is to synchronise and push the Racket files to Github at the same time I fix and add to the API app, which I also let slide lately.

All of these things consist of many small pieces. I can and will do all these things at the same time, one piece at a time.

If you need anything, like a particular API function for a script or app you're writing, just let me know. I'm happy to prioritise any particular piece anyone needs.

As always, questions welcome.


I had a few older message show up in my inbox again.

posted by rob05c: 186 days ago