Infochimps Blog - Organizing Huge Information Resources

Yay! Infochimps is going to Hadoop World 2010. Watch out New York! I (flip) am giving a talk titled “Millionfold Mashups” — I’ll talk about how we store, process and analyze massively numerous datasets and datasets of massive size.

We’re going to order propaganda stickers to give out, and we want to get your feedback on which to print.

Favorites? Terrible puns of your own to add? Want us to send you a set? Let us know in the comments!

• Live Fast and Leave a Beautiful Corpus at Infochimps.org
• Where Hot Singles come to Dataset
• Upload Yours.
• Hadoop-de-doo for you
• Dammit, No, the Other NLP
• I’m Consistently Available. Want to see my Partition?
• Intoxication by Miners is OK at infochimps.org
• Fit your Curves at infochimps.org
• Head in the Clouds?
• Expose your Bits at infochimps.org
• Support Vector Machines!
• Free Variables
• Everyone at our Datacenter has a Nice Rack
• Bayesians Against Discrimination
• Map Reduce, Map Reuse, Map Recycle
• PAXOS in our time
• Pro Axiom of Choice
• Big Chimpin’
• We have the most Cunning Linguists
• P = NP
• P != NP

Several of the slogans shamelessly stolen from this protest by CMU Machine Learning researchers, which I love so much it hurts.

For data geeks interested in the developing world, few places are more compelling to gather numbers about than China. This owes much to its legendary economic growth, the staggering size of its population and global footprint, and hybrid political system. But there is another, often overlooked characteristic of the country at work here: its relentless pursuit of what it calls “scientific development”, which emphasizes the use of scientific research as a means to achieve social harmony and balanced economic growth, has led to an explosion in data-fueled, science-based policy. As a result, China is now one of the largest and most sophisticated data-gathering entities in the world.

There’s a good reason for this. Unlike China’s early post-revolution cadres, the ranks of China’s top leadership today are brimming with scientists and engineers, including President Hu Jintao, who has a degree in hydraulic engineering. When government “works”, these technocrats steer Chinese policy down a painfully cautious course based on five, ten, and even twenty year plans crafted to satisfy discrete social, economic, and technological benchmarks. At any given moment, the country is teeming with pilot projects spanning areas like subsidized housing, health care, industrial development, and family planning, which will ultimately be scrutinized by the country’s National Reform and Development Commission for use at the national level.

This science-based approach is exactly why China has recently come forward with ambitious carbon emissions targets–global warming has a direct, significant impact on its population, and therefore social stability. None of these projects could be completed without good data, and China knows it.

While we’ve been emphasizing social media data with recent posts, we hope to shine more light on the state of Chinese data and bring more of it into the repository in the near future. To this end, and as a special holiday treat, we’re releasing a visualization of major Chinese websites we scraped this past October during the country’s meticulously executed 60th anniversary of its founding. We find the bright colors and flashing lights to be particularly seasonally appropriate.

Click here for the visualization

Our friends at Freebase are having another Hack Day in San Francisco this July.  It’s only two weeks away now and the remaining tickets can go fast, get involved http://blog.freebase.com/2009/06/26/two-weeks-til-freebase-hack-day-sign-up-now/.

Learn about the many cool things that Freebase is doing with their data, and the tools that can be built using their platform.

On a side note, http://infochimps.org/ has gotten a facelift.  We’d love feedback on it: info@infochimps.org.   We hope your browsing experience is better, and we will be happy to roll out new features soon!