Comments on: Linking Bollywood http://silentideas.com/wordpress/2008/10/01/linking-bollywood/ Ideas that make a difference Tue, 07 Sep 2010 00:42:03 +0000 http://wordpress.org/?v=2.6 By: Kumar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-90 Kumar Fri, 14 Nov 2008 06:11:42 +0000 http://silentideas.com/wordpress/?p=133#comment-90 sorry friends for this late reply. I was caught up in some work. So as Tushar mentioned let's first decide on our source of information. If we try to crawl on one particular website for information, the firewall may declare us a spam and restrict access (this came to my mind lately). However dumb servers may not notice. But while gathering data we don't intend to create traffic for the host server and application. Hence need to design the crawler so that it appears to be coming from different ip and at some interval rather than running a loop. sorry friends for this late reply. I was caught up in some work.
So as Tushar mentioned let’s first decide on our source of information. If we try to crawl on one particular website for information, the firewall may declare us a spam and restrict access (this came to my mind lately). However dumb servers may not notice. But while gathering data we don’t intend to create traffic for the host server and application. Hence need to design the crawler so that it appears to be coming from different ip and at some interval rather than running a loop.

]]> By: Tushar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-89 Tushar Thu, 23 Oct 2008 01:33:28 +0000 http://silentideas.com/wordpress/?p=133#comment-89 My few hours of research and reading have clear my understanding of Web Crawler. In fact there are many web crawlers code and 'how to write' strategies available on Internet. My first phase of Web Crawler design Step 1. Source of the information (URL): Who (web pages) are going to be the initial source from where we (Crawler) extract information. Your suggestion required! Step 2. Identify the container and design a method to access it’s contents : Web pages comprises of different "containers" - tables, charts etc (What’s etc??…I don’t know other than these:)),. After that Crawler parse the information and retrieve information of interest Step 3. Data or Information Processing: Information will be processed by Crawler (Information on Internet leads to other Information via link or more specifically hyperlink. We will decide how to process link it in next phase.) Step 4. Network connections: sockets Once we have Crawler ready that retrieves data, we start discussing how to present contents like Digg Arch at the initial level. And then we move to next phase of Crawler design - there is still more to it :) What do you think? My few hours of research and reading have clear my understanding of Web Crawler. In fact there are many web crawlers code and ‘how to write’ strategies available on Internet.

My first phase of Web Crawler design
Step 1. Source of the information (URL): Who (web pages) are going to be the initial source from where we (Crawler) extract information. Your suggestion required!

Step 2. Identify the container and design a method to access it’s contents : Web pages comprises of different “containers” - tables, charts etc (What’s etc??…I don’t know other than these:)),. After that Crawler parse the information and retrieve information of interest

Step 3. Data or Information Processing: Information will be processed by Crawler (Information on Internet leads to other Information via link or more specifically hyperlink. We will decide how to process link it in next phase.)

Step 4. Network connections: sockets

Once we have Crawler ready that retrieves data, we start discussing how to present contents like Digg Arch at the initial level. And then we move to next phase of Crawler design - there is still more to it :)

What do you think?

]]> By: Kumar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-88 Kumar Wed, 15 Oct 2008 05:22:57 +0000 http://silentideas.com/wordpress/?p=133#comment-88 There are two aspects of any project that we take up. Learning and Business I think we start with "learning". But cautiously. We should not reinvent the wheel. We'll start with something that's unexplored and learn while designing and developing it. And when I say unexplored there will be market and business for it. Just that we need to work around it to make it available. So how do we start? where are other guys? why aren't they contributing? Come on people let's look beyond. we need to slog for these things, but at least giving a bit will help.And one more thing, this is not restricted to our group only. There are two aspects of any project that we take up.
Learning and Business
I think we start with “learning”. But cautiously. We should not reinvent the wheel. We’ll start with something that’s unexplored and learn while designing and developing it.
And when I say unexplored there will be market and business for it. Just that we need to work around it to make it available.
So how do we start?
where are other guys? why aren’t they contributing? Come on people let’s look beyond. we need to slog for these things, but at least giving a bit will help.And one more thing, this is not restricted to our group only.

]]> By: Maulik http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-87 Maulik Tue, 14 Oct 2008 15:48:32 +0000 http://silentideas.com/wordpress/?p=133#comment-87 It's not that much silence dont worry. Me and Tushar had long discussion on this topic last weekend. And we felt that we should move further on this idea. Now I like your reasoning and idea about application that leads to discovering new information. Similar idea is implemented in websites like Amazon. Amazon generally tries to figure out what kind of stuff you like and what you might like and it throws to you all related item based on that. In my previous comment what I wanted to figure out was what we really want to accomplish with this project. Are we targeting a particular audience or are we doing a complete experimental program which may or may not be audience oriented? And it would be great if we can come up with a system that can be applied to any information system that is similar to bollywood. But then again we should be clear on what we ultimately want from this project. It’s not that much silence dont worry. Me and Tushar had long discussion on this topic last weekend. And we felt that we should move further on this idea.

Now I like your reasoning and idea about application that leads to discovering new information. Similar idea is implemented in websites like Amazon. Amazon generally tries to figure out what kind of stuff you like and what you might like and it throws to you all related item based on that.

In my previous comment what I wanted to figure out was what we really want to accomplish with this project. Are we targeting a particular audience or are we doing a complete experimental program which may or may not be audience oriented? And it would be great if we can come up with a system that can be applied to any information system that is similar to bollywood. But then again we should be clear on what we ultimately want from this project.

]]> By: Kumar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-86 Kumar Tue, 14 Oct 2008 07:31:34 +0000 http://silentideas.com/wordpress/?p=133#comment-86 why is there so much silence?? where is everybody?? silent !!!!!!!!!!!! why is there so much silence??
where is everybody??

silent !!!!!!!!!!!!

]]> By: Kumar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-85 Kumar Fri, 10 Oct 2008 05:14:41 +0000 http://silentideas.com/wordpress/?p=133#comment-85 Tying up with a website/organization to get the data is surely the straightforward way. But that needs agreement and lot of talks which means lot of time. Why not to use data that is available on the internet. We ask the website for using their data in this way and give them the application as a promotional item(with limited feature) and that works as an enough incentive to use their data. When you are looking for something specific at that time getting data straight is necessary. But when you are just browsing with an intention to discover something at that time navigation and how you present the data becomes important. Anyways we can have different views. But to start with let's have a small bit of code grabbing the data from web. Tying up with a website/organization to get the data is surely the straightforward way. But that needs agreement and lot of talks which means lot of time. Why not to use data that is available on the internet. We ask the website for using their data in this way and give them the application as a promotional item(with limited feature) and that works as an enough incentive to use their data.

When you are looking for something specific at that time getting data straight is necessary. But when you are just browsing with an intention to discover something at that time navigation and how you present the data becomes important. Anyways we can have different views. But to start with let’s have a small bit of code grabbing the data from web.

]]> By: Maulik http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-84 Maulik Fri, 10 Oct 2008 00:39:02 +0000 http://silentideas.com/wordpress/?p=133#comment-84 Alright here is my opinion. We had discussed about this application in the past, but not much. I think it's a really great idea because there is also a business in this area. Two questions, is writing crawler the efficient way to fetch data? How about if we make a flash application and make a partnership with some big website? Just some ideas on that front. I really love what people do with flash and digg. But I think application like this (the one you posted) is not meant to be for largest audience possible. It asks for more brain usage than a normal person would like to use. In my experience there are times when I just need information direct on my face, I don't like to interpret it from different representation. And that's why imdb works really great or <a href="http://www.boxofficemojo.com/alltime/" rel="nofollow">boxofficemojo</a> which uses simple table representations. So you get to think which works for the most number of people instead of what is really a clever idea if you are getting into commercial application. Alright here is my opinion. We had discussed about this application in the past, but not much. I think it’s a really great idea because there is also a business in this area. Two questions, is writing crawler the efficient way to fetch data? How about if we make a flash application and make a partnership with some big website? Just some ideas on that front.

I really love what people do with flash and digg. But I think application like this (the one you posted) is not meant to be for largest audience possible. It asks for more brain usage than a normal person would like to use. In my experience there are times when I just need information direct on my face, I don’t like to interpret it from different representation. And that’s why imdb works really great or boxofficemojo which uses simple table representations. So you get to think which works for the most number of people instead of what is really a clever idea if you are getting into commercial application.

]]> By: Kumar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-83 Kumar Thu, 09 Oct 2008 16:12:53 +0000 http://silentideas.com/wordpress/?p=133#comment-83 ok if you are comfortable with perl can you write a script which can extract particular data from a set of webpages which have same kind of html layouts/structuring. Take for example different pages of movies at musicindiaonline website ok if you are comfortable with perl can you write a script which can extract particular data from a set of webpages which have same kind of html layouts/structuring.
Take for example different pages of movies at musicindiaonline website

]]> By: Tushar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-82 Tushar Thu, 09 Oct 2008 04:16:47 +0000 http://silentideas.com/wordpress/?p=133#comment-82 Because you have an idea how crawler will be; you might have thought of the most suitable scripting language for it! From my side, I think perl will be good candidate! Because you have an idea how crawler will be; you might have thought of the most suitable scripting language for it!

From my side, I think perl will be good candidate!

]]> By: Kumar http://silentideas.com/wordpress/2008/10/01/linking-bollywood/#comment-81 Kumar Tue, 07 Oct 2008 04:21:04 +0000 http://silentideas.com/wordpress/?p=133#comment-81 well than here we go. tushar which scripting language are you comfortable with? Take one of that and write a crawler as I said. I don't how it is done. May be will search on internet a bit. we can then host the app on silentideas itself in the lab section. what say?? well than here we go.
tushar which scripting language are you comfortable with? Take one of that and write a crawler as I said. I don’t how it is done. May be will search on internet a bit. we can then host the app on silentideas itself in the lab section.
what say??

]]>