I just converted user data to SQL. This is a big change, please watch for bugs, and make a post tagged #bugski (or a github issue) if you see any.
This was the last big SQL migration. There are a few small things not in SQL, and a number of fields that need de-duplicating. But this was the last big destructive change.
If you've been following #devski, my plan hasn't really changed, but to reiterate, with this last big SQL migration done (assuming nothing explodes), everything else can be done in small pieces. Which are, roughly:
1. API logins. With user data and password hashes in SQL, implementing API logins should be simple. Which let us serve private data like mail, and your personal feed.
2. API pages. I prototyped this some months ago. This mostly means making SQL views to get all the data for a particular page, and the API endpoint to serve it. The view is the hard part. Theoretically, if we make views and endpoints for every page, we can make a static site of pure HTML and Javascript that simply queries the API. Or you can. Or a mobile app.
3. Data de-duplication. There's an absurd amount of duplicate data, everything from badges, to 'user_sharing' and 'user_shared_by', to 'muting' and 'muted_by'. Racket/SQL functions need added to e.g. get the posts badged by a user, and the users who badged a post, from a single "badges" table. And the duplicate fields and tables deleted. As most of you know, duplicate data is the bane of my devski existence.
4. Converting Arc to Racket. With the last big file loading functions in Racket/SQL now, it's much easier to convert Arc functions to Racket one at a time, from the bottom up. We could skip this entirely, if we decide to fill out the API and make a static site. Or I may do them anyway, at the same time. Because they bother me.
5. Get rid of the last files still used. Cookies, tags, probably a couple more. So that all Hubski data is in a single SQL database. This helps the other tasks listed, as well as making our lives easier, making backups easier, making it easier to write a completely new app from scratch. It makes a huge number of things easier if everything is in a single database.
6. Synchronise github. With all the Arc/app work I've been doing, I've written a lot of Racket that isn't in the github repo yet, just because I haven't had time. My plan is to synchronise and push the Racket files to Github at the same time I fix and add to the API app, which I also let slide lately.
All of these things consist of many small pieces. I can and will do all these things at the same time, one piece at a time.
If you need anything, like a particular API function for a script or app you're writing, just let me know. I'm happy to prioritise any particular piece anyone needs.
As always, questions welcome.
This is exciting. Thanks Rob, you are doing it! You are a man of your word.
Sounds like a good move, though I don't know how the site's data was structured before (though from experience I'd hazard a collection of different and interlinked pieces...) Having a single database for everything can cause its own issues too, particularly when scaling enormously. Happy to discuss "data as a service" and other concepts if that will help - one thing at a time though!
Serialised s-expressions. We're about three orders of magnitude from that problem. We've considered SQL-as-a-service; at some point, it may be cheaper and faster than using a VM host. But it'll almost certainly be SQL. Our data is relational (like 99% of data), and we'll probably never see the scale where it becomes infeasible. Facebook uses SQL with peaks of 13 million transactions/second. We're currently on the order of 1 transaction/second.how the site's data was structured before
particularly when scaling enormously
Happy to discuss "data as a service" and other concepts if that will help
Thanks For Resolving this issue, For more health tips visit my website: www.saferxstore.com
Thanks For Resolving this issue, For more health tips visit my website: www.saferxstore.com
As others have said, my followed tags are now 0. (I just tested something. I re-followed "gaming" and I now show 1 followed tag. However, my name is in the space, news, askhubski tags. Followed tags look like they are still there but not showing on my profile page and not showing in the tags on other posts. Under the "People" tag at the top of the page, Popular Commenters has a "more..." and no lit of names. When you click that "More..." you get an error message "comment failed to load." Commenting in the weekly music thread I get the https://hubski.com/r link and a 502. The post went through. Edit: got a 502 on this post as well. Edit 2: did not get the 502 page when editing this post the first time.
I'm not mad about it or anything, but I've lost all of my followed tags. I can fix it on my end though, so no worries. :) Edit: I get taken to the walking monk screen whenever I follow a tag, but when I get back to the front page the tag is followed successfully.
TLDR bug is fixed, fixing the data now. There was a bug with loading followed tags. It should be fixed now, but anyone who made any user changes since last night will have overridden their tags in the database, so they're still gone. But the old file data still exists. So I'm writing a script to re-populate the database field from the files now.
Hang in there. It's always a heavy slap in the face dealing with fallout from problems after deploying a new version.
If you need to look at someone who made a change, look at mine. I re-added a tag, got the "Following 1 tag" on my profile. Showing now as Following 0 tags. Thanks for the work on the backend stuff. As a guy who also works in the shadows behind the scenes, I appreciate you code monkeys!There was a bug with loading followed tags. It should be fixed now, but anyone who made any user changes since last night will have overridden their tags in the database, so they're still gone.
Got an old message (29 days old) that appeared as new dunno if that's related
Got an old message (29 days old) that appeared as new dunno if that's related