Posted on January 23, 2012
You must be saying, “Econinformatics? What’s that?” Econinformatics is “the application of computer science and information technology to the field” of economics (Wikipedia: Bioinformatics), particularly as it applies to the economic analysis of Big Data:
Big data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. (Wikipedia: Big Data)
Note that there are orders of magnitude of difference between “capture, manage, and process” and “analyze.”
This blog will provide a discussion platform for “Big Data” topics relevant to economists.
A recent article in Forbes titled “Big Data — Big Money Says It Is A Paradigm Buster” highlights the economic trajectory for Big Data in industry, and invariably, for applied economic research.
The fundamental issues associated with Big Data are succinctly described by Anand Rajaraman who is the senior vice president at Walmart Global e-commerce and co-founder @WalmartLabs and a professor at Stanford.
“The tools [for Big Data] are very different. Many of the fundamental algorithms for predictive analytics depend crucially on keeping the data in main memory with a single CPU to access it. Big Data breaks that condition. The data can’t all be in memory at the same time, so it needs to be processed in a distributed fashion. That requires a new programming model.”
This can be hard for traditional data users to understand, He watches students attack Big Data problems by creating a sample, but that defeats the value of Big Data with all its potentially informative outliers.
The challenges are not only faced by “students.” These research and analytic challenges posed by Big Data are facing industry and academic researchers in many fields.
Researchers must undergo a paradigm shift in how they attack research with Big Data. I don’t pretend to have all of the answers about Big Data, but by sharing our knowledge and experiences together, we can shorten the learning curve and all do better work.