Welcome to the new JobGravy Blog!

JobGravy is a new internet career service startup based in part on the technology and business efforts of Alluvial Labs, which (along with the original Alluvial Labs Blog) will be a part of JobGravy moving forward.  We are working on technology that will allow people to organize their own internet data footprints into structured information assets that they can use in a career context.  For example, you could with little effort use it to monitor all new job openings that fit at least five of your more unique (less widespread in job market) interests without having to know which five in advance and without having to fill out any resume profiles or questionnaires.  We believe this kind of capability will mean those doing targeted searches can find better leads more quickly and will allow the folks who might otherwise not be willing to invest in staying alert for the potential lifechanger to overcome the hurdles to do so.

The service is in development, with an alpha program being rolled out in the near future.  I welcome you to check out the information about the company and product on JobGravy.com and look forward to hearing your thoughts about it.

Mark Soper

(*) A special thanks to Vera Novakova for designing the JobGravy logo

Paul alludes to comments made by Gautam Godhwani of Simply Hired.  Gautam pointed out that the reason search engines like Google occupy such an important commercial position is because search is such a good indicator of a person’s intent to take immediate action.

Kevin Krim of Yahoo! Hot Jobs later described much room for improvement in techniques used in job search and other job matching applications, suggesting that we can do better at using available information to make the process easier for employers and professionals.

JobGravy answers Kevin’s call for better matching by expanding Gautam’s notion of intent to include a broader set of information about the individual’s interests (i.e. history of intent) in the search process.

JobGravy is under development, with an alpha release to become available in October 2008.

Top 25 Technologies, Programming Languages, Operating Systems, and Medical Treatments occurring in 5,000 tech job postings (*)

The following statistics can be used to approximate the relative popularity of various computer and medical technologies among employers in August 2008, with the caveat that the sampling was not done rigorously and is not necessarily representative of the broader US job picture.

(*) Results based on a sample of 5000 job listings from major, tech-oriented job boards in August 2008.

• Matthew Hurst provided interesting big-picture perspective on the current and future states of social media analysis based on his research experience at Microsoft Live Labs.  I really appreciated his breakdown of the present state into two realms:
  • Content - techniques that analyze the content of a document (e.g. entities, sentiment, topics, etc.)
  • Structure - techniques that analyze the network context of a document (e.g. links, threads, etc.)

  He then described an overall process built around these two core pieces, including mention of a third realm of temporal analysis - it would be nice to get more explanation on this and on the whole present state picture as Matthew didn’t have a chance to go into much detail.  I really liked that he took a complex situation and explained it in straightforward way - one of those moments when someone describes something, about which you’re already familiar, and without introducing a lot of new information, helps you to gain valuable insight.

  His response to my question about where tagging fit into his model was insightful, though I don’t think I’m yet fully convinced about how applicable the first statement is to many applications of tagging.  He said that they have generally found tags to be of little value when applied to documents because the information in the tags is often redundant with respect to the information in the text.  He did mention that that tags are highly valuable when applied to other media types like images (e.g. Flickr’s very successful tagging system) - right on the money.

  Matthew was good enough to post his slides along with a recap.

• Though not related to social media specifically, definitely deserving mention was Justin Langseth’s 1-hour demo of a Clarabridge implementation.  Not to be confused with an end-user demo, this was a live walk-through of the entire process of deploying and then using their text analytics platform against a modest customer feedback dataset, in one hour.  Clarabridge technology, and Justin’s technical presentation of it, are pretty impressive.  These guys understand BI, and seem to have figured out a good way to structure text data such that it works well within the major enterprise BI frameworks, along with the necessary infrastructure and admin tools to hook everything up without a lot of pain.  In my opinion, their current Windows orientation is less than desirable from a technology perspective.  But their platform decisions, like the use of J2EE, suggest that they appreciate the importance of portability long-term.  And if the enthused customers in attendance are any indication, the business value of what they’re doing far trumps this concern.
• Andrew Bernstein, of Cymfony, related insight from his work in helping companies with large consumer brands monitor market sentiment, test-market new products, and implement market intelligence using data sourced from MySpace, Facebook, blogs, etc.  Interestingly, his four-quadrant social media matrix recommends that companies focus their analytics efforts on social networks first with blogs a close second in terms of market intelligence value.  Social bookmarking, perhaps not surprisingly, faired poorly on this list.  Much more surprisingly given its heavy overlap with both blogging and social networking, Twitter ranked in last place as a valuable source of market intelligence.

Note: The following account of BarCamp Boston benefits from the additional time for reflection and inevitable blurring of facts afforded by the week’s vacation that immediately followed it.

BarCamp Boston is an annual gathering of Boston-area technology people, part of the BarCamp series.  This year’s meeting occurred over the weekend of May 17-18th in Cambridge, MA.  It turned out to be an excellent venue for developing entrepreneurial ideas related to web technology, social networking, mobile applications, software development, etc:

Sessions self-organize on the fly, allowing attendees to promote topics, receive feedback, and schedule talks in real time.  The result is that you have good options at any time to find an interesting topic or create one and lead a discussion around it.  This format also seems to have encouraged some experimental, and especially cross-subject discussions (e.g. At the Intersection of Everything, and the data-visualization discussion held by Matt McKeon of IBM’s Many Eyes).  A discussion I proposed on Natural Language Processing & Semantic Web API’s, covering Calais, Textwise Semantic Hacker (& the $1M Challenge), Y!Q, etc. produced some insight on what to expect (an not expect) from semantic technologies, and on suggestions from web developers on what will be required for more widespread adoption of RDF and other technologies for putting content into more machine-processable context.  One major theme: Better top-down tools are needed to make the process of adding context more automatic.  This ReadWriteWeb article contains a good overview of semantic technologies covered in the discussion.

Warm thanks to Shimon and Mike Walsh for making it happen.

Mark Soper

This morning’s Social Media Breakfast in at Ryles Jazz Club in Cambridge included a speaker’s panel focused on hiring and getting hired using social media. The speakers included:

• Stever Robbins, consultant/executive coach and host of the Get-It-Done Guy Podcast
• Aaron Strout, VP of new media at Mzinga.
• Todd Defren, principal at SHIFT Communications
• Chris Brogan, VP of strategy & technology at CrossTech Media

They touched on two main themes:

1. Using social media to help your careeer. They talked about the importance of your internet activity to your prospects of getting hired, and steps you can take to better represent yourself online. Stever Robbins provided a list of guidelines that dispelled some of the “myths” associated with reputation and career in general. Chris Brogan, with a good dose of humor, described how his blogging connected with his career experiences and the importance of the human element in business (e.g. getting acquainted with someone through his Flickr postings before realizing he’s a prominent VC).
2. Using social media to recruit. This discussion was led by Aaron Strout and Todd Defren, who described their own experiences using Facebook, Twitter, blogs, and other channels. The main idea expressed was that several aspects of a person’s online activity are very important in identifying good candidates:
   • what she writes in her blog or microblog (Twitter) and her Facebook and LinkedIn profiles
   • any content about her that comes up in a Google search, including potentially negative material and material written by others about her

The conversation was enlightening. Todd declared that in the PR space, for example, a candidate isn’t likely to be hired or even interviewed without being “well known” on the internet. More generally, the wisdom prevailing as recently as 5 years ago - that one should be careful about making communications visible online - seems to have reversed polarity. Today the greater risk clearly lies in NOT being visible online, with the chances of a potential employer seeing what happened in South Padre last spring break being actually much less if there is significant other material out there in one’s favor.

There were a couple of issue that weren’t well addressed, despite some audience questions along these lines. The discussion remained limited to the PR-oriented purview of many of the speakers (and audience), so the hiring issues discussed were typically those faced by prestigious PR firms. These hires would be expected to be media savvy and these firms would also tend toward elite hiring practices (e.g. “I want to hire the abosolute best 1-2 guys in the field”). The question of hiring other kinds of professionals (e.g. software engineers) was raised, but a solid answer wasn’t really given. The example of hiring Joel Spolsky because of the prominence of his Joel on Software blog suggests either an unrealistic view of the software industry or social media industry myopia. While many software firms would be psyched to hire Joel, every firm must be willing to hire developers who don’t have widely-read tech blogs, and would probably be better served by looking for the “rank & file” contributors to the social media world, the talented engineers who are reading and/or commenting on Joel’s blog. Finally, there was no attention paid to the majority of people not yet actively using social media. Though we might be better off if more of them did, we can’t expect every dental hygienist to have a blog in order to be doing the job well.

These shortcomings in content covered are quite understandable given the goals and audience of the event, and are pointed out as suggestion for further discussion rather than a negative reflection on the event.  Overall the session was excellent and the folks there were very cool.

Thanks to Bryan Person for organizing!

Mark Soper

Top 25 “Technologies” occurring in 3,200 tech job postings (**)
April 14, 2008

Value:
Using NLP from a system like Calais for entity extraction and classification gives us advantages over just using a keyword search-based approach. In this context of trying to understand what technologies employers are looking for, it tells us answers for which we didn’t know effectively how to ask. To produce the list above using a keyword-based trend search (e.g. on Indeed’s Job Trends), I would need to manually enter 25 keywords. More problematically, I would need to find out which 25 keywords to search for in advance. The entity-extraction approach mines for keywords on our behalf, meaning that we can now also attempt to do search in the opposite direction, with information more able to “find” people interested in it.

Concerns:
Important terms can be misclassified, leading to major inaccuracies. For example: Calais doesn’t seem to recognize “python” as a programming language, so it wouldn’t be represented on this list. It seems to be classifying “asp.net”, “ado.net”, etc. as companies rather than as technologies. Many of these examples are easily fixable.

But the general problem is a very serious one. One of the key points of value comes from being able to learn what you didn’t know how to effectively ask (e.g. which programming languages that I’ve never heard of are most popular with employers). If a system like Calais is to solve the problem, we’re counting on it to have the key knowledge we might be missing (e.g. that Ruby is a programming language).

Conclusion:
Calais is a powrful tool for managing unstructured data. With reasonable amounts of supervision, it can yield some pretty amazing results today in terms of machine understanding of text. And with feedback from a growing number of developers using it in real world applications, it looks promising that it will get much better.

(*) 3200 job listings sampled from about 25 internet & software technology-oriented RSS job feeds from SimplyHired, Indeed, Dice, etc. This was not a scientifically rigorous process in any way.
(**) “Technology” is one of the entity types in the Calais Semantic Metadata

Context is king - there seems to be increased emphasis on technology that puts information into context as a means to combat information overload and find alpha.

David Leinweber described how information on a breakthrough cancer treatment could have been been sourced months before it hit mainstream news and financial markets. The key (excerpted from David’s Nerds on Wall Street Blog): Obtain the “meta-knowledge” (e.g. in this case, the relationship between the vaccine name and the company that makes it) needed to interpret news in a trading and investing context.

Much of the work in this area involves mining massive amounts of unstructured data (web documents) to summarize, filter and otherwise analyze, not only to get to the right content, but to help the investor see it the way he or she wants it:

SkyGrid- uses natural language processing (NLP) algorithms on news to track market sentiment, digesting detail to show broader trends.

InfoNgen - classifies and filters news, blogs, etc. to deliver relevant content in a way that’s specific to the investor’s preferences.
FirstRain - extracts patterns from web documents in a variety of ways that fit into the investor’s research process.

The other class of work being done to reduce overload through context involves contributions from individuals who supply information or help to interconnect it:

Wikinvest - users collaborate wiki-style to profile companies, and link them together through concepts (market trends) - features strong charting (wikicharts), a clever way to capture sentiment (bulls & bears), and soon will have quantitative modeling capabilities

Expert networks are heating up - Alexander Saint-Amand gave a compelling presentation of the value that Gerson Lehrman Group have pioneered in this area. Shortly thereafter Mike Gamson announced that a competitive offering, based on the strength of their 18M “attractive demographic” members, is coming soon from LinkedIn. Already-established Techdirt, having shown how expertise on blogs can evolve into an expert community, were also in the house.

Data, increasingly free, is becoming increasingly valuable - paradoxically, oversupply of open data is creating scarcity value within it. David Leinweber demonstrated several cases of this, including information-rich reports freely and easily obtainable from Edgar revealing how drastically Enron’s SEC filing activity differed from that of its peers leading up to the crash.

Thanks to hosts Tim O’Reilly and Paul Kedrosky for this. Let’s hope this kind of conference becomes a trend as well.

Mark

Unstructured Wiki - RivalMap’s Competitor editor is a good example of handling the challenge of integrating unstructured wiki content with interfaces to the important structured data as well.

Customized Views (Workspaces) - the user has access to multiple, customized views of the system. This is a very powerful for macro-organizing at the level of the various roles and experiences a user may have. In RivalMap, the workspaces appear to be owned by a central administrator, but it may also allow users to create and modify.

Information Retrieval and Discovery - RivalMap wraps the following capabilities into a straightforward, consistently available sidebar filter (some also appear usefully across the top on some pages, in popups, etc)

• Faceted Navigation: Relevant items or categories for the primary dimensions (Customers, Competitors, etc.) are all displayed simultaneously with parenthesized counts.

• Flexible Tags & Labels: The user’s ability to apply his own classifications, organization, and description is quite strong, with support for custom tags and possibly other kinds of labeling and annotation.
• Search-box Navigation: RivalMap makes smart use of auto-complete in the search box to allow for pull-down selection in addition to search.

Infrastructure - It’s important to note that the features above aren’t possible without the right information underpinnings. Even without knowledge of the specifics of RivalMap’s system, it’s clear that the data models and relational machinery have to be flexible enough to establish, maintain, and resolve nuanced relationships between a wide variety of objects with unforeseen characteristics.

A defining characteristic of this kind of application is that it combines: 1.powerful collaborative capabilities for organizing, and discovering information, with: 2. substantial collaborative process or workflow. The fundamental engineering trade-off between power and simplicity is critical to this kind of application, and the RivalMap design team appear to have handled this challenge well.

Mark

——-
Reference:
RivalMap.com - Website, Blog, Demo Video, etc.
Techcrunch Article on RivalMap

1. AWS makes sense for developing sites because their services handle a lot of the routine aspects of scaling a site. This isn’t really a new concept - many web hosts offer infrastructure services that let sites focus on what they do. What makes AWS compelling are the pricing and self-service aspects of their business model. The cost of AWS seems to be significantly lower than that of competitive services, and the usage-based pricing is flexible enough that the costs can be managed relative to volume. And there seems to be low administrative overhead associated with the processes for establishing and managing services.
2. Many of the companies using AWS are extraordinary. Most of the startups who presented in Cambridge are doing innovative things from a technology perspective, and several were very interesting from a business perspective as well:
   • AideRSS - AideRSS is solving an important problem: Helping blog readers to find what’s relevant. In less than a year, they’ve built a valuable service that prioritizes content based on popularity and have ramped up to handling millions of posts daily. They’re looking at adding greater personalization in the near future.
   • Praxeon - Praxeon has a very cool technology called Fingerprint that gleans important concepts from unstructured text, allowing health care providers to quickly search, organize, and get insight from medical documents.

The level of goodwill toward AWS among the entrepreneurial community is high. They’re trying hard to incorporate feedback into their design. For example, customers have been asking for database hosting (*), which would make AWS a more complete infrastructure for a substantial class of prospective customers. Overall, Amazon have put together a compelling service and are demonstrating a commitment to working closely with their customers to make sure they succeed.

(*) S3 is an object store, not a database. Apparently database storage is best done today on AWS by running your own DBMS on EC2, which is oriented toward computation rather than large-scale storage. Some companies are hosting databases external to AWS, then using EC2 to access and crunch the data. One of the participants is scaling very rapidly using this using this approach. Performance and I/O of the database host are key in this scenario, as EC2 isn’t likely to be a bottleneck.

Mark Soper