Feed: O'Reilly Radar - Insight, analysis, and research about emerging technologies - AggScore: 82
Big Data is a major theme on the O'Reilly Radar, so we're delighted to welcome guest blogger Joe Hellerstein, a Professor of Computer Science at UC Berkeley whose research focuses on databases and distributed systems. Joe has written a whitepaper with more detail on this topic.
There is a debate brewing among data systems cognoscenti as to the best way to do data analysis at this scale. The old guard in the Enterprise IT camp tends to favor relational databases and the SQL language, while the web upstarts have rallied around the MapReduce programming model popularized at Google, and cloned in open source as Apache Hadoop. Hadoop is in wide use at companies like Yahoo! and Facebook, and gets a lot of attention in tech blogs as the next big open source project. But if you mention Hadoop in a corporate IT shop you are often met with blank stares -- SQL is ubiquitous in those environments. There is still a surprising disconnect between these developer communities, but I expect that to change over the next year or two.
We are at the beginning of what I call The Industrial Revolution of Data. We're not quite there yet, since most of the digital information available today is still individually "handmade": prose on web pages, data entered into forms, videos and music edited and uploaded to servers. But we are starting to see the rise of automatic data generation "factories" such as software logs, UPC scanners, RFID, GPS transceivers, video and audio feeds. These automated processes can stamp out data at volumes that will quickly dwarf the collective productivity of content authors worldwide. Meanwhile, disk capacities are growing exponentially, so the cost of archiving this data remains modest. And there are plenty of reasons to believe that this data has value in a wide variety of settings. The last step of the revolution is the commoditization of data analysis software, to serve a broad class of users.
To get a glimpse at what that software might look like, consider today's high-end deployments. There are a few different solutions, but they typically share the core technique of dataflow parallelism. Legions of disk drives are set spinning at once, pumping data through high-speed network interconnects to racks of CPUs, which crunch the text and numbers as they flow by. High-end relational database systems like Teradata have been using this approach for decades, and in the last few years companies like Google and Yahoo! have cranked up new tools to bring this process to a scale never seen before.
There has been a good deal of cheerleading and smoke-blowing about these two approaches to massive data parallelism in recent months. Setting aside the trash talk, the usual cases made for the two technologies can be summarized as follows:
| Relational Databases | MapReduce (Hadoop) |
|
|
This roughly captures the positive arguments being made today. Most of the standard critiques of each side are implicit in the bragging points on the other side. Neither side's argument is strengthened by their leading player. Hadoop is still relatively young, and by all reports much slower and more resource intensive than Google's MapReduce implementation. Meanwhile, many people equate high-end relational databases with Oracle, which has developed a reputation for being expensive and difficult to manage, and which does not run on massive clusters of commodity computers the way that many of its high-end competitors do.
Nobody in the data management industry is taking this debate lightly, and the safest prediction is that this landscape will change quickly. The lines between SQL and MapReduce have already begun to blur with the announcement from two database startup companies --- Greenplum and Aster Data -- that each released a massively parallel system integrating a full-featured SQL engine with MapReduce. Each touts a different mix of features from the MapReduce column: Greenplum supports filesystem access and removes any SQL requirements; Aster emphasizes their availability and scaling features. Meanwhile, the Hive open-source project from Facebook provides a SQL-like programming language layered on top of Hadoop. There are also research projects at both IBM and Microsoft in this space, which could evolve into products or open-source initiatives.
So where is all this headed? In the short term, the churn in the marketplace should drive a much faster pace of innovation than traditional database vendors provided over the last decade. The technical advantages of Hadoop are not intrinsically hard to replicate in a relational database engine; the main challenge will be to manage the expectations of database users when playing tricks like trading off data integrity for availability on certain subsets of the database. Greenplum and Aster will undoubtedly push to stay one step ahead of the bigger database companies, and it would not surprise me to see product announcements on this topic from the more established database vendors within the year.
In the longer term, if MapReduce becomes an important programming interface for data analysis, there will be a need to standardize its programming interfaces, to provide compatibility and smooth the road to wider adoption. Alternatively, it is plausible that MapReduce is too low-level and restrictive for widespread use, and it will be wrapped into a higher-level query language like Hive or Yahoo's Pig that will need the same kinds of standardization.
An interesting question is how this will interact with the push toward data-centric web services and Cloud computing. Will users stage massive datasets of proprietary information within the Cloud? How will they get petabytes of data shipped and installed at a hosting facility? Given the number of computers required for massive-scale analytics, what kinds of access will service providers be able to economically offer? It's not clear that anybody has the answers yet in this department.
On a separate note, Hadoop has captured the enthusiasm of educators as a "gateway" to parallel programming. Every computer science major at Berkeley, for example, will write Hadoop MapReduce programs in at least one class (though currently they do so in the educational Scheme language.) Other leading schools have similar programs. Meanwhile, declarative languages -- of which SQL is the most successful example -- are experiencing a renaissance in computer science research, as I've written about elsewhere. So in addition to the changes in the market, the production line for these technologies is being filled at the source. This suggests the possibility of ongoing change over the next many years.Or, My Enterprise is Appliancized, Why Isn't Your Web?
I wrote a couple of posts a while back that covered task-optimized hardware. This one was about a system that combined Field Programmable Gate Arrays (FPGA's) with a commodity CPU platform to provide the sheer number crunching performance needed to break GSM encryption. This one looked at using task-appropriate efficient processors to reduce power consumption in a weather predicting super computer. In these two posts I sort of accidentally highlighted two of the three key selling points of task-specific appliances, sheer performance and energy efficiency (the third is security). The posts also heightened my awareness of the possibilities for specialized hardware and some of my more recent explorations that focused on the appliance market in particular got me wondering if there might be a growing trend toward specialized appliances.
Of course, specialized devices have been working their way into the enterprise ever since the first router left its commodity Unix host for the task-specific richness of specialized hardware. Load balancers followed soon after and then devices from companies like Layer 7 and Data Power (now IBM) took the next logical step and pushed the appliance up the stack to XML processing. These appliances aren't just conveniently packaging intellectual property inside commodity 1U blister packs, they are specialized devices that process XML on purpose-built Application Specific Integrated Circuits (ASICS), accelerate encryption / decryption in hardware, and encapsulate most of an ESB inside a single tamper proof box whose entire OS is in firmware. They are fast, use a lot less power than an equivalent set of commodity boxes, and are secure.
Specialization is also showing up in the realm of the commodity database management systems. At last year's Money:Tech Michael Stonebraker described a column-oriented database designed to speed access to pricing history for back testing and other financial applications. In this case the database is still implemented on commodity hardware. However, I think it's interesting in the context of this conversation on specialized computing because it speaks to the inadequacy of commodity solutions for highly specific requirements.
A device from Netezza is also targeted at the shortcomings of the commodity DBMS. In this case the focus is on data warehousing, but it takes the concept further with an aggressive hardware design that is delivered as an appliance. It has PostgreSQL at its core but it takes the rather radical step of coupling FPGA's directly to the storage devices. The result, for at least a certain class of query, is a multiple order of magnitude boost in performance. I think this device is noteworthy because it puts the appliance way up the stack and is perhaps a harbinger for further penetration of the appliance into application-layer territory.
While appliances are expanding their footprint in the enterprise, it seems like the exact opposite might be happening on the web? Maybe the idea of a closed appliance is anathema to the open source zeitgeist of the web, but in any case, the LAMP stack is still king. Even traditional appliance-like tasks such as load balancing seem to be trending toward open source software on commodity hardware (e.g. Perlbal).
I can't help but wonder though, at the sheer scale that some web properties operate (and at the scale of the energy cost required to power them), can the performance and cost efficiency of specialized hardware appliances be ignored? Might there be a way to get the benefits of the appliance that is in keeping with the open source ethos of the web?
If you've ever uploaded a video to Youtube and waited for it to be processed you have an idea of how processor hungry video processing is on commodity hardware. I don't know what Google's hardware and energy costs are for that task but they must be significant. Same goes for Flickr's image processing server farm and I would guess for Google's voice processing now that its new speech services have launched. If the combination hardware and electricity costs are high enough, maybe this is a good place to introduce specialized appliances to the web?
But how to do that in a way that is consistent with the prevailing open source ethos and that still lets a firm continue to innovate? I think an answer might be sort of DIY writ large; a confluence of open source and open hardware that works like an undocumented joint venture based on the right to fork. Think Yahoo and the Hadoop community or JP Morgan and friends with AMQP but with hardware and you get the idea. Such a community could collaborate on the design of the ASICS and the appliance(s) that hosted them and even coordinate production runs in order to manage unit costs. Perhaps more importantly, specifying the components openly would serve cost sharing across these companies while still supporting flexibility in how they were deployed and ultimately, generativity and innovation for future uses.
There are probably a bunch of reasons why this is just silly speculation, but Google's efforts with power supply efficiency might be seen as at least a bit of precedent for web firms dabbling in hardware and hardware specifications. In fact, Google's entire stack, from it's unique approach to commodity hardware, to software infrastructure like GFS, might be thought of as a specialized appliance that suits the specific needs of search. It's just a really really big one that "ships" in a hundred thousand square foot data center form factor.
As I wrote in Daddy, Where's Your Phone?, it's time to start thinking of the phone as a first class device for accessing web services, not as a way of repurposing content or applications originally designed to be accessed on a keyboard and big screen. The release of speech recognition in Google Mobile App for iPhone continues the process begun with the iPhone itself, of building a new, phone-native way of delivering computing services. Here are two of the key elements:
- Sensor-based interfaces. Apple wowed us with iPhone touch screen, but the inclusion of the accelerometer was almost as important, and now Google has shown us how it can be used as a key component of an application user interface. Put the phone to your ear, and the application starts listening, triggered by the natural gesture rather than by an artificial tap or click. Yes, the accelerometer has been used in games like tilt, parlor amusements like the iPint, but Google has pushed things further by integrating it into a kind of workflow with the phone's main sensor, the microphone.
This is the future of mobile: to invent interfaces that throw away the assumptions of the previous generation. Point and click was a breakthrough for PCs, but it's a trap for mobile interface design. Right now, the iPhone (and other similar smartphones) have an array of sensors: the microphone, the camera, the touchscreen, the accelerometer, the location sensor (GPS or cell triangulation), and yes, on many, the keyboard and pointing device. Future applications will surprise us by using them in new ways, and in new combinations; future devices will provide richer and richer arrays of senses (yes, senses, not just sensors) for paying attention to what we want.
Could a phone recognize the gesture of raising the camera up and then holding it steady to launch the camera application? Could we talk to the phone to adjust camera settings? (There's a constrained language around lighting and speed and focus that should be easy to recognize.) Could a phone recognize the motion of a car and switch automatically to voice dialing? And of course, there are all the Wii-like interactions with other devices that are possible when we think of the phone as a controller. Sensor based workflows are the future of UI design.
Cloud integration. It's easy to forget that the speech recognition isn't happening on your phone. It's happening on Google's servers. It's Google's vast database of speech data that makes the speech recognition work so well. It would be hard to pack all that into a local device.
And that of course is the future of mobile as well. A mobile phone is inherently a connected device with local memory and processing. But it's time we realized that the local compute power is a fraction of what's available in the cloud. Web applications take this for granted -- for example, when we request a map tile for our phone -- but it's surprising how many native applications settle themselves comfortably in their silos. (Consider my long-ago complaint that the phone address book cries out to be a connected application powered by my phone company's call-history database, annotated by data harvested from my online social networking applications as well as other online sources.)
Put these two trends together, and we can imagine the future of mobile: a sensor-rich device with applications that use those sensors both to feed and interact with cloud services. The location sensor knows you're here so you don't need to tell the map server where to start; the microphone knows the sound of your voice, so it unlocks your private data in the cloud; the camera images an object or a person, sends it to a remote application that recognizes it, and retrieves relevant data. All of these things already exist in scattered applications, but eventually, they will be the new normal.
This is an incredibly exciting time in mobile application design. There are breakthroughs waiting to happen. Voice and gesture recognition in the Google Mobile App is just the beginning.
Over the weekend, TechCrunch postulated that with a frenzy of election-related activity, Twitter hit its hockey stick moment in late October. The theory goes that Twitter saw a 25 percent increase in U.S. visits from September to October and is thus about to experience the sort of explosive growth that will propel it into mainstream consciousness.
That could well be the case. In the course of researching our new report, "Twitter and the Micromessaging Revolution," we found that Twitter's user base grew more than 500 percent from October 2007 to October 2008. But we were even more interested to discover that the service has enjoyed an usual effect: as more and more people have joined, the percentage of active users has remained constant [updated:] at about 20%. Among active users (those who post at least once a month), approximately 20 percent post daily and about eight percent post more than 100 times a month (not including known bots and feeds). Though web services usually see a drop in the rate of use as lots of tire-kickers come and go, Twitter's steady usage suggests that a jump in visitors during October could correspond to a big increase in regular users. Yesterday, a Twitterer asked Tim O'Reilly why "Twitter and the Micromessaging Revolution" would be money well spent. Here are ten solid things the report provides (most explained in 140 characters or fewer): 1. Highly readable investigation into why Twitter works--and why it's important 2. Growth statistics from Twitter, synthesized with stories and analysis to give you a comprehensive picture of the Twittersphere 3. Examines the entire micro-messaging ecosystem, including the various players, where Twitter fits and some new developments 4. Clever and useful tips on how to integrate Twitter into your business, backed up by stories about how others are using Twitter successfully 5. Why follower counts may not be the best way to measure Twitter influence, and an alternate model, analogous to Google Page Rank, for identifying influential users 6. Why and how Twitter's loose social graph and the default public nature of Twittering make for a remarkably intuitive and interesting social network 7. Exclusive interviews with Twitter founders and leading users 8. Gentle introduction to using Twitter for the newbie 9. Twitter business models and why Twitter data is important 10. Dozens of links to key resources and examples We've also created a free webcast that pulls from the report's Twitter Primer to give you an introductory overview of the service. (The presentation also has tips and examples not included in the report.) Nearly 450 people watched when we broadcast it live last week, and more than 1,800 have viewed it since we posted it to YouTube four days ago. Perhaps we've hit our webcast hockey stick moment?I met recently with Vic Gundotra, formerly Microsoft's head of platform evangelism, and now VP of Engineering at Google, responsible for all their mobile efforts outside of Android. We were talking about Google's mobile strategy and the insanely cool new voice-activated Google search in the Google Mobile Application for iPhone. But what I really want to share is Vic's story about why he left Microsoft. It was one of those "wake up, the future is staring you in the face!" moments that we all experience from time to time, but often ignore.
The story goes something like this: Vic was out for dinner with family and friends. The adults were on one side of the table, the kids on the other. The adults were debating some issue, and Vic said, in response to a question from one of his friends, "I don't know."
His four-year old daughter Samantha, whom everyone knows as "Tiger," piped up from the other side of the table: "Daddy, where's your phone?"
"What do you mean, where's my phone?" She explained that she'd overheard the question. Why wasn't he just looking up the answer on his phone?
Out of the mouths of babes. Vic said that he realized in that moment that the era of the PC was over, and that the future belonged to cloud applications accessed via phones.
Kamla Bhatt was busting my chops about the same subject when I did an interview with her last week for Mint, the Indian business site. "Tim, you don't talk enough about mobile!" she said. "In India and around the world, there is a whole new generation that accesses the internet, and they have never seen a PC. To them, it's all on their phone."
It's not entirely true that I don't talk about mobile. On Radar, we talk about not just mobile, but all kinds of distributed sensors all the time. And "instrumenting the world" has been a major theme in my talks.
But I plead guilty to Kamla's charge: I think about the web as experienced on a PC, and then about mobile as an add on. The tipping point has come; that notion has to flip: if we're trying to get ahead of the curve, we need to think first about the phone, and then think about the PC browser experience as the add-on.
In short, to borrow Accenture's slogan: "Be a Tiger!" She is the next generation. Always remember her question: "Daddy, where's your phone?"
Boston.com has a really thought-provoking article entitled Depression 2009: What would it look like?. The subtitle answers: "Lines at the ER, a television boom, emptying suburbs. A catastrophic economic downturn would feel nothing like the last one."
This is one of those "Duh!" articles that makes you see the obvious. As the article notes:
Most of us, of course, think we know what a depression looks like. Open a history book and the images will be familiar: mobs at banks and lines at soup kitchens, stockbrokers in suits selling apples on the street, families piled with all their belongings into jalopies. Families scrimp on coffee and flour and sugar, rinsing off tinfoil to reuse it and re-mending their pants and dresses. A desperate government mobilizes legions of the unemployed to build bridges and airports, to blaze trails in national forests, to put on traveling plays and paint social-realist murals.
Today, however, whatever a depression would look like, that's not it. We are separated from the 1930s by decades of profound economic, technological, and political change, and a modern landscape of scarcity would reflect that....
Unlike the 1930s, when food and clothing were far more expensive, today we spend much of our money on healthcare, child care, and education, and we'd see uncomfortable changes in those parts of our lives. The lines wouldn't be outside soup kitchens but at emergency rooms, and rather than itinerant farmers we could see waves of laid-off office workers leaving homes to foreclosure and heading for areas of the country where there's more work - or just a relative with a free room over the garage. Already hollowed-out manufacturing cities could be all but deserted, and suburban neighborhoods left checkerboarded, with abandoned houses next to overcrowded ones.
And above all, a depression circa 2009 might be a less visible and more isolating experience. With the diminishing price of televisions and the proliferation of channels, it's getting easier and easier to kill time alone, and free time is one thing a 21st-century depression would create in abundance. Instead of dusty farm families, the icon of a modern-day depression might be something as subtle as the flickering glow of millions of televisions glimpsed through living room windows, as the nation's unemployed sit at home filling their days with the cheapest form of distraction available.
While no one knows for sure just what might happen, the article puts fascinating flesh on how a downturn severe enough to be characterized as a "depression" would be experienced in today's market. The article calls out these likely scenarios:
Desurburanization, and a preference for renting over owning a home;
The re-legitimization of hand-me-down culture (Hey, I never gave it up!), including national second-hand chains;
A preference for reliability over novelty and style;
"The neighborhood appliance shop could reappear in a new form - unlicensed, with hacked cellphones and rebuilt computers";
A fatter society, "as more people eat like today's poor" (not if Obama listens to Michael Pollan; this observation makes Pollan's advice even more compelling with regard to government agricultural subsidies and food policy!)
"More crowded subways and cheap, unlicensed day-care centers" (Read the story for the explanation);
That's far from all. And there's even more great stuff in the comments. Thanks a lot to @hadleystern, retweeting @gregoryng for pointing me to this article. It was juicy enough that merely passing it on via a retweet wasn't enough. I want to give it some blog link love, and a place for people to comment.
What would you have to cut in the event of a serious downturn? How would it change your life and the life of those around you? Answering that question will also make clear what some of the right policy choices are for government, as well as some interesting opportunities for entrepreneurs. Go read the story and the comments on it, then come back here and give me your ideas.
The fifth Where 2.0 Call For Participation is now open. This year we're going to focus on location-ware technologies and their implications. The iPhone and Android have paved the way for a new breed of app and Where 2.0 will be focused on it.
If you want to join us on stage submit your talk by December 2, 2008.
Where 2.0 will be held in San Jose, CA from May 19-21, 2009. You can watch all of Where 2.0 2008 online -- for free.
Some of the topics on the radar for Where 2.0 are:
Location-Aware: We will be exploring the implications of our new location-enabled lives, particularly around mobile phones and transponders. What feature is worth sharing your location?
Reality Mining: With the increase in location data come more macro views of our lives. If you want to know where to go in San Francisco, for example, City Sense will show you which parts of the city are hopping. What does this type of information mean for consumers and the enterprise?
Augmented Reality: The location-enabled phone will become a viewfinder for our world. Your phone will be able to tell you what you are looking at. It will also let you leave notes for the next person. What are other cool projects in the works?
Immersive and 3D Imagery: There's an imagery battle happening and consumers are winning. Our world is being documented to an unprecedented degree. While two device manufacturers acquired the mapping data companies, the internet giants have invested in cameras, planes, and satellites. Where will this take the location industry?
Mapping Tables: It's difficult to collaborate in person with an online slippy map; a paper spread out on a table or tacked to a wall is still better. Digital mapping tables are attempting to beat back paper once and for all. By providing everyone the same view and editing capabilities plus the ability to turn on and off layers, will they be able to do it?
Government 2.0: Governments are treasure troves of data. Increasingly they are releasing it online for free. ESRI's release of ArcGIS has also aided the battle by providing municipalities with this ability. This data is aiding both the citizen and Government agencies. How is this critical information being put to use?
Crowdsourcing: Pioneered by OSM, the rest of the mapping industry is catching up. Let's examine where they are taking it.
Disease Awareness: Our increasingly connected world allows diseases to spread in record time. These same networks alert us to outbreaks. We're going to examine new geocentric approaches to epidemiology.
Cartography: Each map has a distinctive look and feel. What are the trends in design and user experience?
Workshops
Back by popular demand, Where 2.0 will have a full day of workshops where participants can dig deep into a range of issues and leave the conference armed with new tools and skills. Workshops are one hour and fifteen minutes in length and will be held on Tuesday, May 19. Topics we'd like to explore include, but are not exclusive to:
Geo Support in Web Application Frameworks: As people design their own mapping applications, there has been a need for built-in geo support. We're looking for workshops that teach about Mapstraction, Modest Maps, Open Layers, GeoDjango, GeoRuby, MapCruncher, and other tools.
Mapping APIs: The location space would not have gotten as far as it has today without all of the innovation in the mapping API space. How can you test the limits of these free resources?
GeoTargeting: Knowing users' locations has never been more important. Identifying it accurately can be difficult and expensive. What are the best methods?
Privacy Implications: As you are collecting user data, keeping track of your users, or collecting geodata, are you aware of the relevant laws? What would you teach others?
GeoBrowsers: Google Earth and NASA WorldWind are both amazing geobrowsers. How can you get the most out of them?
Data Management: Geo applications work with massive amounts of data. What are the tools, tips, and tricks that can be used to manage it?
Protocols and Formats: GeoRSS, GML, KML, EXIF, Microformats, Geo OpenSearch. Which formats are on the way in and which ones are on the way out?
These are just some of the technologies and transformations we've noticed and represent just the starting point for the program. While we'd like you to tap into the theme as your inspiration in writing your proposal, feel free to wander. What are you working on that will change the world, or at least the world you're in? What project is bringing you pleasure, or teasing your brain? Surprise and delight us; shake us out of our assumptions. We're angling for shorter talks with longer breaks so you'll have more time for one-on-one interactions.
Vivek Kundra, the District of Columbia's CTO, isn't just talking about transparent government and participative democracy, he's working hard to make DC's massive data stores transparent and inviting your participation.
I first heard about Vivek's push for transparency when he spoke at an Intelligence Community Conference in September (I just happened to be speaking on a panel thanks to a twitter-induced serendipitous introduction to one of the conference organizers - @immunity). He was there in sort of "one government entity to another" role to demonstrate that data could be shared and that it is valuable to do it.
I was impressed with the risks he was taking to push hard for the "democratization of data" and for what he was calling The Digital Public Square. What came through really clearly was that he didn't just view this as a technology exercise, but as a way for citizens to participate in real ways and to increase government accountability. It was an engaging and refreshing talk.
It's not exactly news at this point that he came up with $20,000 to offer prizes for the best applications to be built on top of the district's data. After all, the submissions have been long closed and the winners have already been announced. However, I thought it might be worth pointing out that you still have until tomorrow to vote for the two people's choice award winners.
I thought it was kind of fun to just poke around in the list of submissions and see what people came up with. As you can imagine many of them are geo-spatial display's of district stores, but there are some other ideas in there as well. Take a look, see what you think, and get your people's choice vote in.
And just because the contest is over doesn't mean it's too late to build something. Take a look at the catalog of data and see what comes to mind. This is just the beginning (Mayor Nutter of Philly, I'm looking at you...).
One of my favorite sessions at the recent Web 2.0 summit was Tim's half-hour conversation with Shai Agassi, the CEO of Better Place. Better Place aims to make electric cars widespread ("the electric car as the de facto standard") by addressing major issues that have held back electric vehicles: affordability and convenience.
In a relaxed conversation with Tim, Shai described an electric car industry that resembles the mobile phone business. Just as telecom companies sell mobile handsets at a discount if one is willing to commit to a contract, their subscription-based model will allow consumers to purchase an electric car at the fraction of the normal price. Car owners will pay additional fees based on the amount of miles they drive and the type of car they choose to own. To support their subscribers, Better Place will also build extensive networks of charging spots and battery exchange stations. They will build the first "Electric Recharge Grids" in Israel and Denmark.
Prior to starting Better Place, Shai was a president at software vendor SAP. The interview briefly touches on IT and enterprise computing.
[NOTE: Web 2.0 summit videos are available on YouTube.]
Larry Lessig was busy last week: he was in New Zealand for the LIANZA (Library and Information Association of New Zealand Aoteatora) conference, flew back to California to man the phone banks for the Presidential election, and then spoke at Web 2.0 Summit. Fortunately, several of his talks were recorded.
Monday night he gave a public talk at Auckland University, sponsored by University of Auckland Law Faculty, the Legal Research Foundation, and the University of Auckland's Department of Commercial Law. I was in the audience for this, and it was vintage Lessig: culture, remix, legal reform but not abolition. His framing was brilliant, and everyone cheered at the end. The Law Faculty blog wrote about it, there was a great write up on Public Address blog, and there's a clip on Vimeo. I saw official looking video cameras, but I haven't been able to find the online video yet.
On Thursday, back in the US (maybe it was Wednesday back in the US, curse that dateline) Lessig gave a High Order Bit at Web 2.0 Summit. He spoke about corruption, focusing on quickly making the connection between money, trust, and independence explicit. It's short, hard-hitting, and fit perfectly with the "Web meets World" theme of Web 2.0. All the Web 2.0 Summit talks are online at blip.tv, including Lessig's talk. The talks are also available on YouTube including Lessig's talk.
But earlier, on Tuesday, Lessig keynoted the LIANZA conference. He spoke about corruption in the context of libraries, libraries as anchors of trust and independence and the ways in which copyright, like money, erodes trust and independence. It was brilliant, and the video is online (albeit with a Windows codec). It's 1h12m in length, so pour yourself a warm tea.
You even get the question and answer session at the end, in which he addresses New Zealand's latest Copyright Amendment Act which adds takedown provisions for alleged copyright infringement. In no uncertain terms he condemns it. "I don't know if you recognize this but you guys are at the edge of the world [...]. The idea that you would start cutting off internet access to people sitting at the edge of the world is crazy. [...] Allowing these random cutoffs of your connection to the rest of the world to define the future of New Zealand Internet is just crazy, so I hope that Government rethinks this before they march forward with this demand by copyright holders to push you back into the Dark Ages." Amen, brother!
Google.org has released Flu Trends, an online reporting tool for flu-related search activity. It's long been theorized that Google's search data would be useful to predict epidemics. This is the first time they've released a tool like this to the public. As they say on the main page:
We have found a close relationship between how many people search for flu-related topics and how many people actually have flu symptoms. Of course, not every person who searches for "flu" is actually sick, but a pattern emerges when all the flu-related search queries from each state and region are added together. We compared our query counts with data from a surveillance system managed by the U.S. Centers for Disease Control and Prevention (CDC) and discovered that some search queries tend to be popular exactly when flu season is happening. By counting how often we see these search queries, we can estimate how much flu is circulating in various regions of the United States.
This tool comes to us via Google.org's Predict & Prevent initiative. You can download the data for your own analysis.
If you want to check the status other diseases HealthMap, an online "Global Disease Alert Map". The automated site uses a variety sources including Google News, traveler reports, and official WHO alerts to track diseases across the world. It is another Google.org investment
Tools like Flu Tends will work in areas where people have access to the internet or use Google. Though Google is number one in the US, it doesn't have top status in all countries and will not necessarily have enough data to make meaningful determinations. If Flu Trends proves valuable enough I wonder if other countries' CDC-equivalents will pressure their top search engines to develop similar tools.
This morning, via twitter, I came across two contrasting blog posts, one from JP Rangaswami (@jobsworth), and one from Martin Varsavsky (@martinvars), that seemed to me to sum up the very essence of the problem I've been calling out in my "work on stuff that matters" talks.
In Faster Horses in the Age of Co-Creation, JP argues that Henry Ford's maxim, If Id asked people what they wanted, theyd have said faster horses , is wrong, that the essence of innovation today is giving the customer what he or she wants:
That customer knows that part of what she wants is to be able to figure out what she wants. She is both consumer and producer, a partner in the process of co-creating value. The senior partner in the process of co-creating value.
So today, if she asks for faster horses, we dont build her a car. We need to find out whether she meant a roan or a piebald or a chestnut or a bay. When she tries the piebald out and decides she wanted the roan, thats what she gets. Our job is to make it easier for her to buy or rent or lease the horse, to make it safer for her, to make it more convenient for her in terms of where the horse is to be picked up and dropped. To make sure the horse is well, that the riding equipment is securely and safely fastened.
Now, to be sure, JP is making a valid point, namely that in consumer marketing, we need to listen to and empower the customers, not try to ram products down their throat.
But it's unfortunate that JP used the Henry Ford story to make his point, because in so doing he set up a straw man that ignored what Ford was really saying: that breakthrough innovations don't come from market research, even from "web 2.0" market research via deep customer engagement. They come from the singular vision of an inventor pursuing his or her own passion, cutting a Gordian knot that others simply accept as "the way things are."
The Wright brothers would never have gotten to the airplane by listening to customers; Henry Ford would never have sought to put an automobile in every household; Tim Berners-Lee would never have gotten to the World Wide Web; Jimmy Wales wouldn't have started Wikipedia; and Jack Dorsey wouldn't have started twitter, where I became part of this particular conversation.
That's why Martin's post is so poignant, and so important. Where is the Future We Were Promised? he asks:
Five years into the 20th century, Einstein was living his Annus Mirabilis. Where is our patent office today? Who is our Einstein? Are we the first generation in many years incapable of true innovation? And lets not just talk about things as complicated as the theory of relativity. I remember complaining about the drill when I was young, and my dentist telling me that when I was grown up he would have to find another job because we would have a vaccine against cavities. Where is this vaccine against cavities? Where are the cures for catarrh and AIDS? Where is that future devoid of poverty in which robots were going to do everything for people and we were going to dedicate ourselves to art and culture?
Unfortunately, when I look around me today, during the end of 2008, I see humanity leading an unsustainable life based on technology that should already be obsolete. I believe that it is time for us to engage in some serious self-criticism and start to invest in science again, because the list of unsolved problems grows longer every day. If we continue on like this, not only are we not going to have a future, but we are going to end up without a present.
Like JP Rangaswami, Martin overstates the case a bit. I believe that there is some truly amazing innovation happening on the net, in alternative energy, and in life sciences, and that we are going to wake up one day and be blown away by the future we're creating. And oddly enough, many of those innovations will come from harnessing the collective intelligence of all those people that JP says we should be listening to. But it won't just be to give them what they want; it will be to put them to work in new ways, getting them to contribute to an inventor's vision, not just to customize it for their better enjoyment. Breakthroughs in speech recognition and automatic translation, for example, are driven by the data we all contribute; similar effects will soon be felt in personalized medicine, robotics, and many other areas.
But I also believe that the choice is stark: just give people want they want, leading us deeper into a consumer culture whose very financial fabric is wearing thin, or seek out big, hard problems that other people take for granted as unsolvable, and remake the world.
In a talk I attended many years ago, Joseph Campbell said that the Knights of the Round Table were the archetypal myth of Western civilization, the idea that each of us, alone, must go off into the deepest, darkest part of the forest, populated by monsters, on a quest to make the world a better place.
I wrote this piece about a month ago as the Welcome for Make: 16, which will be on the newsstand soon.
As I write this, there is panic on Wall Street despite Washingtons $700 billion rescue attempt. The crisis is not contained by U.S. borders, but extends to Europe and Asia. Like many people, Im incredulous. How could this happen?
Wall Street hired the best and the brightest, paid them handsomely, and gave them unlimited resources and technology. It turns out they were building enormously complicated castles made of sand. A great wave washed them away, astounding all the smart people who devoted their lives to speculation, not production. Their models based on historical data predicted future profits, not collapse. Few people saw this coming until it hit.
It was the triumph of data over common sense, said reporter Adam Davidson on the excellent episode of This American Life called The Giant Pool of Money. Economist Michael Lehmann in the San Francisco Chronicle called it the triumph of ideology over common sense. Its obvious both common sense and the common man have taken a beating.
Its hard to stomach that our government must bail out Wall Street. It really means weve bet our future on the same people who created the present situation. To paraphrase a joke Ive heard: Its like going to a casino in Vegas and rooting for the house. One New York Times reader expressed the frustration that many feel: Why cant we take half of the $700 billion and just build something?
These events shake our belief that free markets work to the benefit of all. The fundamental tenet of capitalism is the invisible hand : Adam Smith wrote that by pursuing his own interest [each person] frequently promotes that of the society. This year, Nobel Prize-winning economist Joseph Stiglitz said: In this sense, the fall of Wall Street is for market fundamentalism what the fall of the Berlin Wall was for communism it tells the world that this way of economic organization turns out not to be sustainable.
A headline in the Christian Science Monitor says: With finance crisis, hands-off era over. Government will need to be more assertive in regulating Wall Street. But I think it goes beyond that. I wonder if we, as individuals, have been living in our own era of hands-off. Have Americans become so disengaged that weve become dependent on some invisible force to provide what we need? Have we gotten used to leaving important matters to experts, until they turn out to be wrong?
Isnt it time for us to become hands-on again?
We, the people, face enormous challenges. Apart from the economic mess, we know fundamental changes are coming because of global warming. Our dependence on fossil fuels is not sustainable. Change is coming, whether we want it or not.
Better we meet the challenges head-on rather than hide. New York Times columnist Thomas Friedman summed it up: We need to get back to making stuff, based on real engineering not just financial engineering. We need to get back to a world where people are able to realize the American Dream a house with a yard because they have built something with their hands, not because they got a liar loan. ... The American Dream is an aspiration, not an entitlement.
We have to believe it starts with each of us not some faceless government or corporate bureaucracy. Its time for us, individually and working together in business, to reconsider what it means to be productive, not just profitable. Its time for us to reengage in how our government sets priorities for education, health care, housing, and transportation.
The DIY mindset celebrated in this magazine must again become an essential life skill, rooted once again in necessity and practicality. Our future security lies in knowing what were capable of creating, and how we can adapt to change by being resourceful.
A challenge this great can bring out the best in us. We need everyone, because every person has something to contribute. We need a showing of all hands.