Lance Vick, Founder of Tawlk, explains social search and advocates open source. Lance shares a bit of the process of deciding which bits of code need too be protected and which bits get released to the community. He goes on to discus examples of sharing code that has helped him to make a better product with a fairly small team.
Links referenced in the show:
The music in the show, Have Mercy — Big Walter Horton, was provided by Mevio’s Music Alley.
Transcription
-
Lance Vick 00:00:22 So, I am Lance Vick. And I am lead engineer and founding member of Tawlk. And Tawlk is a new type of social search and analysis engine. So you can put in any keyword and we can show you how that keyword is being used across the whole social web and what people are saying about it, how people are feeling about it, and some useful stats to get an idea of what’s going on with it.
-
Ryan 00:00:47 Well, that sounds pretty cool. I think that there’s a lot of buzz around that. You know, search was big. And now people are saying kind of social search is next. I think it’s kind of ambiguous as to what that means. But people feel like there’s a certain quality of information that comes from -- whether it be kind of looking at your social graph or just the way that things are kind of spreadable.
And then all of that information seems like it’s higher quality for whatever reason if you’re searching through Twitter, searching through Facebook. Maybe it’s more authentic. Maybe it’s less about people trying to sell their own things, I’m not really sure. But what’s been your experience with that?
- Lance Vick 00:01:18
Yeah, so certainly found that when doing research if you search for, you know, a major car brand or something and you search in a search engine like Google you’ll often get stuff that is either trying to sell you -- you know, the car brand website, dealerships, things like that. You may not necessarily be getting an idea of how people actually feel about it or whether it’s negative or whether it’s positive or any of those sorts of things.
Whereas when you search in social media you’ll get people being much more vocal. They could be saying I hate X and this was a terrible experience for me, or this was an awesome experience for me.
So I feel like in social media you’ll often get a much less biased view into what’s going on with the topic.
- Ryan 00:02:05
It seems like when people look at social search, for the most part they are talking about either a tool for the marketing department, or just, you know, ego stroking. Is that kind of where you’re at? Or, it seems like there’s also big value for a user to get that information that you talk about.
- Lance Vick 00:02:25
Sure. I think it can be very important for pre-purchase decisions.
So like one thing that we have online right now is some comparisons. And we’re of course going to be rolling out a lot more of these, but this sort of sets the tone for what can be done. Like we’ve got all the major cell phone companies side by side ordered according to how influencial they are in social media we are right now, how many people are complaining about them, et cetera. And so you can sort of see that ranking and line-up if you’re going to make a buying decision. Oh, hey, this company is dead last right now and everyone is complaining about them, maybe I want to stay away from them, and take a look at maybe one of these companies that’s closer towards the top of the list.
So I feel there’s a real market there for pre-purchase decisions -- and, you know, you’re just getting an idea of what other people are saying about something before you put your money down on it.
- Ryan 00:03:17
That seems mind-numbingly hard to think about just seeing what people say and being able to actually parse that our and figure out -- even just is this a positive statement? I mean, there’s a whole lot of sarcasm on the internet. I’m not sure if you’ve heard.
- Lance Vick 00:03:31
Oh, yeah. We’ve certainly run into this in doing sentiment analysis. And all the research that we were doing building our own sentiment analysis system. We actually found that pretty much everybody else has embarked on this has a lot of problems with that as well.
It’s a difficult problem to solve well. It’s certainly not a problem that can be solved 100%. In fact, about the highest we’ve seen anyone get is about 80% and that’s where we’re sitting out now too
When we started looking at lots and lots of social texts, we find somebody will be, like, I love my job :(. Which is clearly sarcasm.
So we have found, though, that emoticons do tend to really set the tone. And that whenever we can get stuff with emoticons in it, that makes for amazing training data, because that’s pretty much somebody that’s tagging the data for us. And if you get enough sample data, then the machine can get smarter and smarter. So we have a machine learning system.
So in our case, to build our classifier, we feed in a couple million tweets, about a million negative, about a million positive. And so when you get that many it starts to learn probabilities of words and things. So you end up with certain things floating to the top.
And so we knew it was starting to work when at the top of the negative list for most exclusive negative we saw FML. And you think about that, there’s not really a lot of positive situations you can use FML. So we agreed with that.
And then on the top of the positive side, we saw one we didn’t expect, which was GBU. And none of us even knew what BGB was and we looked it up. And it was God bless you. And I’m like, oh, OK, that’s a positive thing. And there’s not really a situation you can use that in a really negative context.
And so getting a big massive word list and basically scoring them based on their probability of being negative or being positive
And so using techniques like that with a huge word base gets you around 75%, 80% most of the time. And so with enough samples -- say you’re searching for a topic and you’ve got, you know, a few thousand samples, that margin for error, even with it there, you can still get a pretty good idea of what most people are saying whether it’s negative or positive.
So when -- in this area, it’s just getting as far as you can, enough to be able to get an idea of what’s going on.
- Ryan 00:05:45
That is amazing that you’re using emoticons as some sort of sarcasm Rosetta stone. I don’t know if I like that or not, but that’s definitely fascinating
- Lance Vick 00:05:55
Yeah, we certainly found that most of the time whatever the emoticon is, is sort of somebody tagging the sentiment for us. And so you can get a whole lot of training data just by searching for happy face or searching for sad face. And then when you strip the emoticons out of the text and then have it go through again, words that end up being used sarcastically are words that show up close to as many times, or a similar amount of times on both th negative and the positive context.
And so words that are used sarcastically a lot aren’t weighed as highly as words that are almost exclusively negative or almost exclusively positive.
So, like the word bad actually isn’t necessarily a super negative word, it’s not a very strong word, because people use it in positive context, like that’s, you know, badass or something. We just weigh based on how much is used and what context it’s used in.
- Ryan 00:06:42
You just gave me this image of all these regular expressions like, "well, if bad is bad unless followed by ass…
- Lance Vick 00:06:53
Yeah, the Bayesian tree is a bit crazy. I’m very thankful that we generate that. I wouldn’t want to write it by hand.
- Ryan 00:07:02
This is interesting also because I think that we need a certain sohpistication in how we parse this stuff, becuase I would like for the most part hashtags to go away, right, especially the way people use them in marketing. I think that they’re fun. Like, I like to use them as sort of jokey elements, right, but like this idea where people say, hey, please talk about this thing and make sure that you use a certain hashtag or whatever. What’s your thought, you know?
- Lance Vick 00:07:27
Well, I feel like as natural language analysis gets better that, yeah, hashtags won’t necessarily be as important or necessary. So many times, like I’ll go to conferences and it was an astounding difference. I went to BarCamp Miami, where they advertised the hashtag really big everywhere. Everybody used the hashtag. And as a result there was way, way more data that you could actually find about that event.
And then I went to BarCamp Orlando where they actually had triple the attendance, but they didn’t have any particular advertised hashtag. And so people were mentioning, like, you know BarCamp or BarCamp Orlando, but since there wasn’t any consistency there, it was really, really hard to group the data together so it was really difficult to find related data. And it seemed like overall people didn’t talk about it as much because of that, I guess, confusion or lack of call to action.
So they definitely work well as a call to action, I think. And having something specific that people can parse out. So I understand, you know, the need for them today, but I feel like as the machine learning systems get better. And even as Twitter’s own search, and they’ve got their own hoodoo cluster and everything, gets smarter, then I can figure out that, OK, this person is from Orlando and they just mentioned the word BarCamp and a lot of other people are mentioning BarCamp today, so we can be smart about just grouping all those together whether or not there’s a hashtag.
So I do think there’s going to be more of a trend in that direction to where the hashtags won’t necessarily be as important, because the system kind of overcomes that. But we’ll see
- Ryan 00:09:01
Yeah, I guess I probably sounded a little bit more negative about it. There’s definitely value in it, but it does seem like it’s sort of stop-gap hacky thing. I mean, especially even if you think about where they came from, right, it was just sort like we needed this need to add extra metadata, but when it turns into these marketing campaiagns, right, where it’s like some company slogan and they just keep kind of tagging it on things and they ask you to do that as well for some reason or another…
- Lance Vick 00:09:23
Yeah, totally lame.
- Ryan 00:09:24
Yeah, like what -- how is that possibly helpful? It’s like even if they do manage to get trending, like, what does it do, right?
- Lance Vick 00:09:32
Yeah, I definitely agree that there certainly wholly abused right now by most. And so that’s made them a lot less attractive than maybe perhaps they ought to be. So…
But like anything marketers can abuse just about anything. So, that’s the nature of the beast.
- Ryan 00:09:50
Yeah, I mean, any time there’s a system, somebody is going to try to game it for some -- and not necessarily a bad thing, right, that’s what pushes -- pushing those boundaries is kind of what makes things evolve.
- Lance Vick 00:10:00
It’s true.
- Ryan 00:10:01
You talked about machine learning. So that implies that you’re a huge company, right?
- Lance Vick 00:10:04
Um, you would think so. And it seems like most of the companies that deal with machine learning are certainly much -- much larger. But we’ve actually built really lean systems that are flexible and can run on very small infrastructure. In fact, our primary service for sentiment analysis can run on a personal computer if you’d like.
And so we’re actually not a huge company. In fact, I’m the only full-time software engineer right now. And we also have a part-time software engineer in Canada. And I’m in Orlando. And we’ve got a couple of guys who help out.
So right now we’re a very, very small company. We’re a startup. So we’ve gotten a lot done with very little.
- Ryan 00:10:47
That’s impressive, because when you just think about projects like this, they were not something a small team could do before. It was -- people would have to strive to be part of a big company just so they had the resources.
- Lance Vick 00:10:56
Yeah. And today, you know, you don’t really necessarily need huge resources to do that. You know, we’re getting this done with an extremely small team, because we’ve got resources and open source resources that exist like Redis and LTK and different libraries that do a lot of the heavy lifting so you can think more about the logic and what it is that you actually need to do. And systems that were run on a single computer, or even a single personal computer that before would take, you know, a giant server cluster and you’d have to have a huge investment.
And so I think in our case we have such a small team and small resources, it’s actually forced us to find really efficient ways to do thing, because otherwise they weren’t going to be able to get done at all. So it’s certainly been interesting.
- Ryan 00:11:46
You talk about open source. And you said kind of the more common thing that people talk about open source. I would like to dig in a little deeper, becuase I know that your relationship is kind of more complicated than that, but what you said first was without a lot of resources we could use open source stuff kind of implying that thank goodness there’s some stuff that doesn’t cost anything, right? But it’s more than that for you, right?
- Lance Vick 00:12:10
Yeah. In our case, because we obviously leverage open source -- as I just mentioned, we leverage it quite a bit. But we want to make sure that we are giving back in kind. And that we’re giving unique things that we’ve come up with and problems that we’ve solved back to the community so they can make use of them, they can improve them, do crazy things with them that we’ve never even thought of. And so it’s definitely a give-take we feel.
- Ryan 00:12:38
Cool, so when you talk about what it is that you give, do you find yourself putting a lot of code back upstream into some of these larger projects that you do? Or is it mostly just projects of your own that you let people dig into?
- Lance Vick 12:50
Certainly I have plenty projects of my own and things like that and so does, you know, other engineers, but as a company we actually put out four of our major libraries in open source. And so we actually have our official company Tawlk GitHub account which has all those on there. For instance, the sentiment analysis and the data collection libraries are currently open source.
So we couldn’t really find a good open source sentiment analysis solution, so for us that was a chance to say, well then let’s make one. So it may not be the best implementation in the universe, but you have to start somewhere and people can improve it and it helps everyone.
- Ryan 00:13:35
Now that’s interesting, because sentiment analysis it sounds like it’s pretty core to your business. And that’s one of the things that you’re like, oh, it seems like a lot of times you know maybe someone will open source the CMS that drives the marketing, right, but not necessarily the actual app that they do. Is that as close to your business as it sounds? And if so, why would you share that?
- Lance Vick 00:13:56
Well, first off we have a lot of different services out there that say they do sentiment analysis and they even go as far as to say it’s 90% accurate. When I see stuff like that, I’m instantly saying I call BS, nobody has a system that’s that highy accurate. They’ve done tests and they found that humans only agree on sentiment like 80% of the time. So at some point you’re disagreeing with humans anyway.
And so when you have companies that say things like that, but they have absolutely no transparency at all, you really don’t know if there’s sentiment analysis system is reasonable or even works. So with ours, we feel we can be tranparent, people can test it. We can say not only are we saying these things, but by all means pull down the source code, check for yourself, make sure that we’re not just giving you a marketing line.
But also the sentiment analysis itself is one piece of a larger equation. We have our Tawlk scoring system. That is a proprietary algorithm. That’s patent pending. And that is based on sentiment popularity and the overall influence of people to talk about a topic. And we kind of actually take all of those together in one big algorithm and make our Tawlk score.
So that is sort of our secret sauce in analysis. And sentiment is just one piece of that, that we can, you know, make modular and repurposeable for other things.
- Ryan 00:15:08
Cool. What I’m hearing is, it gets really close to it, but you’re not really giving away all the secrets. You’re just really focusing very honestly on where the value comes from. And it’s a much smaller set than maybe people might think of when they see the whole project.
- Lance Vick 00:15:23
Sure. We’re definitely trying to make data collection better, trying to make it standardized. We’ve made our own formats for normalizing data from lots of different social networks which is really important before you can analyze it. And sentiment analysis becuase we feel ther’s a very wide range of uses on that. And things that people will do with that that we haven’t even thought of yet.
And so we certainly want to be able to leverage and encourage creativity and let people, you know, use it for whatever they want. So we certainly feel there’s a lot of value in making these open for everybody.
And it kind of goes to an article I read about a founder of GitHub. And he says open source almost everything, which is, you know, open source everything that is repurposeable, is modular, that doesn’t allow people to build your product and then have everything that’s unique to your product as your own module, as your own implementation that you can, you know, roll forward and make money with. And that way you kind of suit the business side of making money and you suit the open source side end of giving back to the community and encouraging and fostering more innovation.
So, we definitely say we agree very strongly with that.
- Ryan 00:16:34
Yeah, it makes a lot of sense to me. Hopefully it makes the sense to a lot of the listeners, but it’s good to hear.
It’s interesting to dig down a little bit more, because you say you want most of this stuff free to everyone. There’s a certain discrepency between what you say, or at least seemingly so, and I think that we should dig into that a little bit too. And you’ve got patents and you’re really active in open source. I think some people see that as conflicting views.
But I don’t think it has to be, but I think that a lot of time people say that, right, it’s either you’re with us or against us, right. Which side are you on?
- Lance Vick 00:17:09
Yeah, and I’m actually one that is really against software patents when they prevent innovation. In fact, I would say -- at least for me personally, my partners would probably disagree with me on this in some respects, becuase they’re more business focused -- but at least for me perosnally the only reason I see for having patents is to defend yourself when somebody is trying to take away your ability to innovate. So you have a lot of time where even open source projects are coming under fire -- obviously, the whole Oracle/Android lawsuit.
So having leverage to defend yourself is really awesome and makes you more attractive to, you know, companies that might want to buy, because you’ve got these protective kernels.
So in our case, we have things that are patented that don’t even touch the open source libraries themselves directly like our Tawlk score algorithm and our overall search and display methods. So our particular way of doing things is covered in the patent, but not at all, you know, restricting what people can do with the open source library. So I guess we’ve made sure to have that distinction there
- Ryan 00:18:12
So what kind of, you know, following on GitHub do your projects have? Do you have a lot of involvement, or have you seen realy cool uses of it out there that just make you think I am so glad I opened this up?
- Lance Vick 00:18:23
Yeah. Already, we’ve had some interesting stuff.
First of all, our hyve library, which is a library that allows you to do social media data collection actually in the browser so you can actually make the collection portion a distributed problem. We’ve had a fair amount of code committed to that and people that have jumped in and rewritten functions. And I look at the code and go, that’s fantastic. I would have never thought to write it that way. And it runs so much better now. So we’ve definitely had a fair amount of that.
I’ve also seen a couple projects people put out there. One guy recently was showing me a project he did with the hyve which was collecting social media data on the fly and then actually doing topical analysis on it as the data is coming in live. And so he was basically searching three different topics, having all the day spew together and then having a browser based topical classifier actually start splitting them back up into three different topics again. And it gets smarter and smarter the longer that you run it.
And I’m looking at this going that is absolutely fantastic, you’re doing topical analysis at a browser level. And so that’s an experiment I really haven’t had the time or even the idea to really spend time pursuing. But this guy did. And I was able to instantly see the value in that.
And so I certainly see it as we get more exposure to these projects, people will be able to be empowered to do all sorts of things. So…
- Ryan 00:19:51
That’s kind of the beauty of an open source project is suddenly your team is not quite as small as it was.
- Lance Vick 00:19:55
Yeah, exactly.
I don’t think really any one company could ever hire as many people to innovate as you can when you open source something. And people that are working for all sorts of, you know, maybe big companies around the world in their weekend time, or spare time can still contribute amazing code to your projects and key insights. And so you get a much wider pool of experience pouring into the project.
- Ryan 00:20:20
Yeah, it’s interesting how that whole economic norms and social norms where when people are trying to join together as a community to solve a problem they’re far more willing to just give you stuff as opposed to: otherwise you could not afford it.
- Lance Vick 00:20:35
Yeah. Yeah. You know, it used to be when everything was super proprietary, you know, it could be extremely expensive to embark on a new project. And most of the time well outside of what may be a lot of time budgets a lot of projects have, especially researchers. And so that’s actually one big goal of ours is be able to empower researchers.
Say you’ve got a really cool idea that you want to research, you don’t have to go have, you know, thousands of dolars to throw at software. Here’s some libraries to just run with it. And now you can focus on your idea.
And so it’s getting to where these days anything you can really think of doing is probably a decent open source effort out there to help you with it.
- Ryan 00:21:14
I guess even more than that what I meant was specifically like that developer’s time. Like the ones that have added to your project I would say with a startup I would guess, I’m not sure the financial situation, but if you were trying to bid out some of these enhancements. I’d say a lot of them you’d say, kind of doesn’t make sense at this time to spend money on that, right? And they would have to bill enough that it would probably not make a lot of those make sense for you.
But the fact that they’re just kind of solving their own problem and kind of leveraging your technology and so free with what it is that they’re doing, now all of a sudden you’ve got all the benefits, right?
- Lance Vick 00:21:45
Oh, yeah, absolutely. Yeah.
So things that I would consider. Like, you know, of course I’ve got my set development plan of what can be done that we have to focus on as a team. And I can all these other blue sky ideas that I simply just don’t have time -- that could, you know, drastically help the project. But I just don’t really have time to explore. And so, yeah, when people can go pull down the code and go try this, you know, crazy idea and spend a couple of weeks working on it. And they throw up some source code and prove, hey, this actually works. And then I can go, oh, well had I known that was going to work, you know, maybe I would have budgeted my time towards it, but now I’ve got this amazing code that now I can reintegrate and use that can, you know, really improve our projects.
This one guy is working on improving, you know, the browser based sentiment classification system.
So eventually instead of having servers classify these sytems, we can actually, you know, move a lot that to the browser.
- Ryan 00:22:43
Wait, in the browser? What -- that -- doesn’t -- sound intuitive to me. How does that work?
- Lance Vick 00:22:51
As I was mentioning before we have not a lot of infrastructure. And so already we actually pushed a lot of the work to the browser, even like the data searching and collection. The majority of that actually happens at the web browser level. And then your web browser is actually searching a lot of data and then sending it back ot our server where we run sentiment analysis on the server and then echo it back to you and tell you, hey, this much of the data that you were searching was negative or positive.
And so by actually being able to do even the sentiment analysis at the browser level, well then our servers that much more work is taken off of them, and which means we can scale that much further. Because when you can make each individual browser do the majority of the work for themselves, the majority of the unique work for the unique data that their searching, all of a sudden one server is now, you know, capable of handling a tremendous number of clients more than it could otherwise, if that makes sense.
- Ryan 00:23:50
I mean, it does that you say it, but it isn’t like a way to solve the prolblem, it seems like the common approach, right. It seems like whenever your doing a lot of number crunching the idea is you want to push that back to a server somewhere and not do it in the browser. So that’s interesting that you’re leveraging that in that way.
- Lance Vick 00:24:05
It is something that we ran into quickly. We actually kind of started this path because of the API limits. And we found out that you’ve got tons of services like Facebook that only give you like, you know, 600 requests per 10 minutes, which if you’ve only got one user in your system that’s probably reasonable, but if you’ve got thousands then the first couple are going to get results and then nobody else does.
So by actually pushing all that to the browser we found out that, you know, then now everbody have API limits to themselves, becuase you have all these many IP addresses that are collecting data.
And so it used to be a big thing of doing everything on the server, because really the people’s desktop computers weren’t very powerful. But now the smart phone in my pocket is more powerful than, you know, every computer that I grew up with, you know, all my first computers anyway, combined. And so definitely in a different world where the CPUs behind the web browsers are certainly tremendously more powerful. And we’ve got a lot more play there. Javascript engines are much, much faster now. In many situations Javascript is executing at near the speed of C code.
So if I can just ship a Javascript library to soembody’s browser and give them a piece of the work while they’re using the site without affecting their resources, then it’s a win for them, becuase they’re getting an awesome free service. It’s a win for us because with our thousands of users all taking small pieces of the work we don’t have to have thousands of servers, which lets us scale and give away things for free that we otherwise couldn’t afford to.
So it’s actually worked our really well for us.
- Ryan 00:25:38
That’s actually really interesting. I like how it all comes down from you had to work against a constraint. And then that kind of forcing you to shift the way that you think about a problem ended up having benefits that seem almost completely unrelated, right. Because I think otherwise if you didn’t have those API limits would you have gone down this way?
- Lance Vick 00:25:56
Well, probably not. In fact, because it seems like the way that most people are solving this problem, and also why social analysis tools contributes to them being so expensive, the conventional way of getting around that problem is to just get lots and lots of servers. And so you may have, you know, an Amazon EC2 Cloud or something with, you know, 1,000 nodes all working on collecting different topics from -- so that way they’re spread across different IP addresses. And that’s a very expensive infrastructure and you need a lot of investment to even be able to build, you know, such an infrastructure
So, yeah, in our case it was sort of either, either we find a solution for this problem or we quit here. So we built the hyve library. And the hyve library actually runs completely browser side and does all the data collection. So that was sort of our solution for that. And, you know, we opened sourced that. So people can improve it and add more services right now. I think it supports like 8 services, but if somebody wants to add GitHub support, for instance, they could. So overall it’s been a big solution all around.
- Ryan 00:26:57
That is really, really cool. I like that. Kind of how the serendipity of how you came to that.
So let’s see we talked about hyve mostly. You’ve got a lot of projects though. Let’s drill down real quick, before we run out of time here, what kind of things can people find in the Tawlk repos?
- Lance Vick 00:27:15
So, in addition to hyve, which is, you know, the browser collecting, we also have kral which collects data in the same way and is the same data normalization, but it’s intended to run on the server. And that’s something that you’d want to run for more sandbox data that you need to collect, you know, 24 hours a day without any breaks in it. So that’s more intended for, you know, long-term data collection.
And then we have synt, which is our sentiment analysis library, which is also intended to run on the server. So you feed it thousands of examples of negative and positive. And then you start feeding it new samples and it will tell you, you know, how it classifies and what’s the percentage of chance that it’s negative and the percentage chance that it’s positive. And so we wanted to make a simple library and framework for thabecause we really couldn’t find any sentiment analysis libraries out there in the open source world that weren’t just like a list of bad words and a list of good words and weren’t exactly very helpful
And then a new project that we’re spending some time on is this broswer based sentiment classification system that actually takes a lot of the wok that we put int synt and being able to spread that out to the browser. And that’s the one I was talking about a minute ago. We’re starting to get some help on and some great ideas for.
So, those are sort of the four core efforts that we have going on right now.
- Ryan 00:28:30
Sounds great. It sounds like you’re really solving, you know, problems that are kind of unique to now, which seems like that would be very rewarding. It sounds like you’re working with tools that you very much enjoy using. It just sounds like you’ve got a pretty good setup going here.
- Lance Vick 00:28:43
Well, we’re pretty happy with it. It’s something I get to work on every day, you know, I feel great doing, becuase I know that I’m being able to give some awesome tools back to the community and also building a unique product that can be, you know, profitable and, you know, help us grow as a company and hire more engineers and take some of that time to make even more open source code. So everbody wins.
- Ryan 00:29:05
That’s, I mean, such a -- such a nice note to leave off on. We should all be so lucky.
So, if you want to check out the product, that’s at Tawlk with a W -- tawlk.com, right?
And if they want to check out your code, they go to github.com/tawlk
- Lance Vick 00:29:24
Yep. That’s where it’s all at.
- Ryan 00:29:27
Very fun. Thanks so uch for talking to me about this.
- Lance Vick 00:29:28
Thank you.