Sunday, October 14, 2012

Research sites / searchable databases

Really, any area of interest can turn into a searchable database.  I should really be able to get this kind of stuff done in my sleep by now.  Specifically, I was thinking of DIY hardware - sourcing things like sensors and so on.

The basic process would be:

  1. Get a basic semantic picture of the domain (i.e. what items are of interest? What attributes distinguish them? Who is active in the area? Who blogs about it?)
  2. Crawl the Web by means of link following, search term identification, and so on.
  3. Find the valuable data sources and determine the database schema that best encodes them.
  4. Scrape data on a periodic basis into your database, and make sure you have keyword-rich databases available.
  5. Provide some kind of commenting/forum functionality.
There are, of course, lots of this kind of thing out there - they all suck.  A small community of devoted fans can make or break this; you can become the definitive guide to a very small domain.  Monetize with ads and with sales.

No comments:

Post a Comment