a thoughtful web.
Good ideas and conversation. No ads, no tracking.   Login or Take a Tour!
comment by mk
mk  ·  3429 days ago  ·  link  ·    ·  parent  ·  post: Most data isn’t “big,” and businesses are wasting money pretending it is

Unless you deal with large datasets, I think it's difficult to understand just how difficult designing meaningful analysis becomes. In the end, when it comes to market analysis, you'd probably do just as well going with the 'gut feeling' of someone that has demonstrated a keen understanding of the market. Otherwise, mining data like user behavior is probably only going to allow you to tweak things around the edges. The problem being, you might learn two important things, and in reacting to them, work counter to a third and possibly more important thing that you didn't even know to look for. -That's why I think A/B testing is most often bunk.

As an aside, I just started changing the way tags are stored, and have been trying to anticipate what information might be best to store with them so that we can use them more effectively. I can't anticipate using a separate machine to crunch these data any time soon.





forwardslash  ·  3429 days ago  ·  link  ·  

I've been wondering: how is data persistently stored on hubski?

mk  ·  3428 days ago  ·  link  ·  

Magnetically. ;)

Hubski has no database. Instead, data is stored in flat files as s-expressions (lists, it's Lisp!). All the data you see here is actually in memory, loaded on startup, plus newly added data. Tags have existed as elements within post and user data, but I am currently creating an independent directory for them, so they can have their own associated elements that can be updated, sorted more quickly, etc.

There's definitely some work to be done with all this as things progress. It should be fun.