Zach Beauvais

Journalism Needs Data in 21st Century

// August 5th, 2009 // Comments // perspective

|This first appeared as a guest post on ReadWriteWeb, republished with kind permission

Journalism has always been about reporting facts and assertions and making sense of world affairs. No news there. But as we move further into the 21st century, we will have to increasingly rely on “data” to feed our stories, to the point that “data-driven reporting” becomes second nature to journalists.

The shift from facts to data is subtle and makes perfect sense. You could that say data are facts, with the difference that they can be computed, analyzed, and made use of in a more abstract way, especially by a computer.

With this mindset, finding mainstream data-driven stories doesn’t take long at all. A quick scan of the Guardian’s home page tells us that swine flu cases are up by 50%, according to “fresh figures…[that] will be released this afternoon.” The story here is that we’re in danger because swine flu is on the rise. Reporting the current figures available for swine flu alone wouldn’t be all that interesting. The news comes from comparing the current figures to last week’s, which is a very simple form of data analysis. By making use of published data and running one’s own analysis (and building on the analysis of others), we get something very news-worthy indeed. It moves the definition ever so slightly, from “saying and asserting” to “analyzing and publishing.” But it obviously works only for data that is accessible.

There is nothing new about pointing out the importance of public data being made available. Sir Tim Berners-Lee has discussed at length the importance of governments and institutions putting their data online, making it accessible and useful. His TED talk and interviews with ReadWriteWeb and Talis (disclosure: I am a blogger at Talis) all explain his belief that by publishing linked data we can begin to solve many of the problems the world faces. Innovations in medicine, science, and development could all be achieved if only currently hidden data were made available. Data-driven journalism could be the first step in realizing this dream. The best stories would then come from innovators who read about trends reported in news media and are then able to draw new conclusions and solve bigger problems. In his recent discussion with BBC, Berners-Lee said that the next step is to go for low-hanging fruit by just getting the data out there.

Thus far, this has made a lot of sense to me, and I have been tracking the publication of linked data and increasing access to public knowledge as emerging trends over at Talis. But my perspective has shifted a bit in the past few weeks.

First, there was data.gov and President Obama’s call for more access to government data. A sitting head of state (and one of some significance) was clearly calling for public access to government data: this was news! But the idea has been discussed, praised, and debated for a while since then and may have lost some of its luster.

Then about a month ago, UK Prime Minister Gordon Brown made it part of his digital strategy to prioritize the publication of government information. He asked Sir Tim personally “to help us drive the opening up of access to Government data in the web over the coming months” and appointed Berners-Lee an official governmental adviser. By now, neither of these stories is news and comparisons between the initiatives have been made.

The Guardian newspaper recently launched its own Data Blog, with the intention of letting readers access, mash up, and reuse much of its information in the form of data, which could in turn drive stories.

What is perhaps not as explicitly recognized is the voracious appetite for data that has been apparent for months. It is less about turning good ideas into stories and more about seeing how data informs our understanding of events happening right now. Each new initiative is another piece of low-hanging fruit picked.

Access to data is important: it drives innovation and even social change. Governments that publish their data have to become more transparent. Humanitarian organizations that make their findings known could spark bigger projects and source innovative solutions from their communities. Scientific findings and raw information could be used to solve bigger problems than the result of a single experiment or trial could ever manage. Even the simple comparison of two or more facts can lead to new insight, and all of these things happen only when the walls around an institution become porous.

2009 could become known as the year of data, the year of open access, or the year of the semantic Web (see links above for how this relates), and it may also be the first year when it becomes news that data wasn’t published in a story when it should have been. That a government body isn’t being transparent or is blocking access by publishing its findings in PDF or other non-linking formats would make a very interesting story indeed. We can expect to see more and more organizations and public bodies remove their own barriers through initiatives and legislation. Examples have been set, and seeing excuses die along with barriers is not far-fetched.

Do you know of other data-driven stories? We’d love to hear about any insights that were made through publicly accessible data or where this data might come from next.

Guest author: Zach Beauvais is a Platform Evangelist for Talis and editor of Nodalities Magazine.

Related articles by Zemanta

• Sir Tim Berners-Lee talks to the BBC’s Rory Cellan-Jones about Linked Data (cloudofdata.com)
• How Open is ‘Open’ ? (cloudofdata.com)
• Interesting semantic web stuff (derivadow.com)

Democracy and the Web: the UK gets it while America tries to control it.

// July 25th, 2009 // Comments // blogging, emergent, interesting, perspective

I read yesterday that twitter has been banned from the White House. In the post, Marshall Kirkpatrick joked that this could be a reason we haven’t seen much from Obama’s twitter stream recently. I must admit however, my initial reaction was sympathetic with the White House for pragmatic reasons, though the attitude of the Press Secretary’s attitude towards “twitterers” did raise the hackles. It makes sense to be secure in the White House, to make sure people aren’t saying things which could be dangerous or cause scandal through carelessness. “Loose lips sink ships”, don’t they?

But I think there is a wider idea here, which I think I’ve glimpsed between the lines.

twitter is used around the world to announce what we eat for breakfast. I use it to pass on little observations, like you might to a room full of mates, when there isn’t anyone there to share with directly. News of Michael Jackson’s death reached me via an off-colour joke on twitter. These are uses for a technology which it would be difficult to commend.

However, I also use twitter to share news. When my grandfather recently passed away, I received dozens of messages of encouragement and sympathy. Several of us here in Shropshire organise a monthly get together to network and discuss tech-trends and the work we do through the @shropgeek. Important announcements at work, and shared interest groups often rely on twitter for their spread and response, and I’ve had customer service reps from big companies personally respond to my feedback. Of far more significance, the government of Iran was unable to stop twitter, allowing its citizens to tell the rest of the world what was going on when all other forms of communication were censored, blocked, or monitored. And citizens from around the rest of the world responded.

twitter is a medium, and suggesting someone is petty and fatuous because they use it is like suggesting everyone on TV is unimportant or vain. There is no connection between the inane talk-show host and the investigative journalist or head of state! The point is in the message, not the vehicle. But, the point this raises in my mind is that twitter, and other forms of web-enabled channels make for a high level of transparency, and I don’t think the US (in particular) is a power who likes transparency at the moment.

This has lead me to question what I think of the use of web-media by politicians and important figures, particularly in the US and UK. It makes me wonder whether the “Loose lips” philosophy is misguided in the modern world. You see, closing channels, blocking communication, and monitoring messages suggests a democracy that doesn’t trust its citizens with the truth. Sure, there are controls and securities which must be in place, and I’m not suggesting for a moment that every clerk should have a constitutional right to twitter state-secrets…

…or am I?

You see, the United States is global super-power on par with, and probably only directly comparable with great states of the past called Empires. The notion of an emperor of the US would chill the blood of most of my family, and thinking of the US as anything but a “democracy” is practically heresy. After all, citizens’ rights are ensconced in the very foundation myth and history of the US itself. “We hold these truths to be self evident, that all men are created equal…” begins a letter to a despotic monarch, sparking off the touch-pad for liberating men from the rule of figureheads, class and social bondage. But part of this very myth* that the Republic, “of the people, by the people and for the people” should be based on citizenship trusted to look after themselves and even take up arms to defend their status as such citizens. There is a deep-running notion in the American psyche that if the government were ever to get too big for it’s boots, it is the right—nay, duty—of her citizens to act to reduce her power.

But I don’t believe the US is a state that can be corrected by its citizens very effectively at all. I think of the complexity, size, presence and byzantine nature of the US Government, and I feel disconnected, small, and powerless to change anything. Over the past decade, in the name of security, Americans put up with reductions in liberty, and I think this principle is bleeding through the cracks in the facade of governance. Blocking channels is saying: “we don’t trust you”.

What would a country look like where the public had access to the vast majority of government information? Where government officially made use of the media its citizens used? Where government officials were held accountable via the various media whenever they were caught being mis-represented?

Oddly, I think it’s the UK.

The “traditional” media here are a powerful force. It is seen as a near human right to have intimate news of public officials and dealings, and watching politicians being interviewed by members of the press is like eaves-dropping on a job interview or witnessing a cautious father’s first meeting of a prospective suitor for his only daughter. The press is a force to be reckoned with here, and it’s not seen as the trustworthy force itself, but is is composed of citizens, not officials.

Does this scandalise the government? Yes… and no. There is a very different attitude toward elected officials here, which doesn’t translate easily to American. A Member of Parliament is legally referred to as “Honourable Member”, but the “honourable” is not taken as read by the average Brit. For decades, for example, Members of Parliament have enjoyed a system of expenses whereby they can claim for nearly every cost of living: from second homes to food and utilities. The true level of this feeding-trough has recently been blown wide by the press (who subsequently have been enjoying their own self-congratulatory feast, but that is another post.)

But, I think the UK get the web, probably because it’s used to dealing with powerful media. I follow Number 10 Downing Street (the metonymic residence of the Prime Minister) on twitter. No 10 doesn’t say much, and I don’t think it’s going to expose any state secrets, but I like the fact that it’s there to be engaged. The Office of Public Sector Information (OPSI) hosts an important blog which outlines the government’s plans to expose public data for normal, every-day citizens to have a play with and to see what’s going on.

And, in the last few weeks, the Prime Minister himself has turned himself around almost 180º in my personal opinion. He represents a party for whom I have less time than either of the other major contenders, and I’ve rather lazily accepted him as an incompetent oaf. But, he’s finally earned my pint and invite to dinner, if not my vote (if any of his secretaries are reading this, just tweet your acceptance, and I’ll find some pheasants and Pinotage). A few weeks ago, he appointed Sir Tim Berners-Lee to a Parliamentary advisory role with the explicit intention of opening up and pushing public data online. This is a major point, because it leads to transparency through public accountability. There seems to be a movement for Parliament to see public data as belonging to the people who bought it with their taxes, and this seems to be the most democratic way to see it. His recent TED talk also made me think he’s got a lot more to say than he perhaps has to date; though I think many of his points raise more questions than they answer.

Much has been said in the online world about the new American administration’s use of social media and the web to mobilise supporters during the election. But there hasn’t been much since. President Obama launched data.gov, but there is very little data there. I think the web is seen as a tool for messages, as a part of a bigger campaign, and as a security breach. It’s something to be used with your own agenda, and only under one’s own strict parameters. And, perhaps most non-democratically, it’s been used to broadcast and to cajole—It has not been used to engage. The fact that social media have barely been touched since the election could point to a wider attitude that citizens only matter for the brief time they’re required to vote.

The UK has already exposed much of its public data, and it’s planning to publish more and more as Linked Data (machine-readable, immediately useful resources), and it’s made plans to be more open, grasping the web and the transparency it’s brought through the hard lessons that spin is impossible with a well-informed citizenry, and on the Open Web, there is less room for your own message than there is for humanity.

*mythos is greek for “story”, and it is from that perspective I use the word myth: that it conveys the notion of a commonly-held understanding, not that it is entirely fantastic or without truth.

Jacqui Smith—going, but not forgotten

// June 3rd, 2009 // Comments // Topical, perspective

At lunch time today, I learned from a tweet that UK Home Secretary, the Right Honorable Jacqui Smith will resign as soon as convenient to the Prime Minister. Most sources seem to cite the now cliched “expenses scandal” as the main reason, but also mention various moments of Home Secretarial chagrin.

My initial reaction was one of elation. “Finally, and not too soon!” is a sentence too easily thought about someone I think has done more to damage the UK than any one holding power in the last decade.

But this is not how I want her to be remembered. I do not want her to have gone because of a media-fueled feeding frenzy of shame for her expenses or her husband’s use of public funding to watch porn.

Maybe I should clarify?

Last February, I wrote a piece about the Home Secretary which still attracts concerned comments. It told my story of how her tough stand on immigration affects the legitimate, the law-abiding, and the defenseless rather than dealing with any perceived threat by illegal immigrants, migrant workers, or European nationals.

Jaccui Smith should resign because she used a misleading public story to back plans which bring shame and financial hardship to people.

One of the most controversial campaigns Ms Smith backed was one to introduce compulsory ID cards to the UK. She claimed that people would willingly sign up to such a scheme, and much talk of fighting terrorism with these bits of plastic ensued. This is a lie, really. People don’t want it—or, at least, the majority of people don’t seem to want to spend their money on them. Oh, yes. Not only is the scheme hugely expensive to the public, but individuals will have to buy them themselves. The card could cost £30, £60 or £93, depending on which confused source reported the cost [1]. The scheme itself will cost the UK taxpayers around £5bn with an additional £375mn coming from foreign nationals, who will be the first victims of the programme.

The ID cards information produced by the Home Office itself reads, to this linguist at least, shockingly like propaganda. It produces press releases such as this one, which appear to show an eagerness which belies the fact that nearly every mention I’ve seen in media regarding the scheme has contained the word “Controversial”. It has proven so controversial, indeed, that the latest report I read showed that the Home Office seems to have changed its mind massively. This leads me to believe that the Home Office misleads, both in content and in linguistic implication through its own media.

Jacqui Smith should resign because her policy to introduce ID cards is not democratically supported, is questionable and expensive, and her promotion of the scheme is biased and untrue.

On any number of profiles trying to make sense of or summarise Ms. Smith’s career, another low point has been the proposal to allow imprisonment without charge for 42-days of suspected terrorists. This has proven controversial (simply search for “Jacqui Smith 42 days” to see.) and demonstrates Ms Smith’s disregard for the ancient principle of due process. The Proposal was mercifully scrapped last year.

Jacqui Smith should resign because her principles do not include thousand-year-old fundamentals of basic citizens’ rights.

Not everything undertaken by Ms Smith has been terrible. Throughout a mixed career, it should be mentioned that she held her ground, proved resiliant, and handled difficulties well (until now). The Right Honourable Jacqui Smith, MP has been the UK’s first female Home Secretary, and no doubt worked very hard. However, this was expected of an MP, and a Home Secretary should handle her job well, which is why it is news when this has not been the case.

Ms. Smith has been forced out of office following news of the least-damaging scandal. Shame on the media for blowing out of perspective the theft of several thousand pounds in claimed expenses in the light of a proposed spend of £5bn to encumber citizens and ostracize legitimate foreign nationals. Shame on the reporting that claims her husband’s silly use of £10 to watch porn should take precedent for misleading a nation, trying to create and enforce draconian laws and damaging the reputation of a great democracy.

[1]: It should be noted that the £93 claim is old, and that the Home Office seems to say most recently that the cards will cost £30 for early adopters and rise to £60 after two years.

Additional ID Card Stories include: Scotland refuses to back them., and some initial concern about the cost raised by the LSE.

Image: “Jacqui Smith at anti-binge drinking launch” by “photographyjournal” via flickr

Twitter metadata—metaphor?

// February 10th, 2009 // Comments // Semantic Web, emergent, interesting, perspective

This post featured originally in Nodalities Magazine.

I’m sure I’m introducing old friends; but Twitter is a “microbloggiing” platform, to give it its proper description. it gives users 140 characters to publish status updates, comments, gripes, complaints, praises, news and whatever comes to mind. It’s burst out of its original answer to the simple question: “What are you doing?” and users often tweet just about everything.

One interesting innovation is the integration of the hashtag. Simply a hash symbol (#) and a tag descriptor for the comment. This gives people the ability to follow particular threads of updates or participate in conversations around an interest. They’re often used, for example, to update the goings on from conferences (#FOWA for example). People give their own content this little bit of information, and a search engine can find them. People can add additional information and follow conventions which allow for distributed trends that anyone can follow and interact with.

The recent snowfall in Britain gave rise to a flurry of tweets about road closures, amounts of snow falling, schools closing down and all the other chaos unleashed. When users followed a simple convention, however, this information got organised. People quickly adopted the #uksnow hashtag to track the topic; and eventually someone worked out a way to capture all the info needed to follow these geographically. By tweeting the first half of a UK post code plus a rating out of ten snowfall, anyone following the thread knows exactly where it’s snowing. It’s like an instant weather polling station, distributed across the country. It can go a step further, however, when services can actually mashup these tweets when users turn their simple status updates into a mini line of code.

This little bit of information allows for people to write software to track and automate the twitter information. This interactive map from benmarsh.co.uk, for example, actually plots a visual graph of snowfall across Britain. Bigger snowflakes indicate larger numbers out of ten in the poll. It’s simple, really. Ingenious, possibly. But the fundamental distinction between this tracking ability and the noise of thousands of Twits shouting about the snow is that little bit of metadata.

So, is this use of twitter a metaphor for the Semantic Web? It’s certainly a picture of automating information flow using metadata via software. Sounds Semanticcy to me.

 

God’s business strategy?

// October 10th, 2008 // Comments // emergent

Ever had a superficial conversation suddenly run your blood cold?

On the short cab-ride to my hotel following FOWA (more to follow), the cab driver was explaining the extension of the conference venue. This is not usually the preamble to a conversation that changes your perspectives, except maybe on the architectural uses of steel spider’s legs and concrete. However, he was saying that the folks that bought Manchester United (actually, he should have said City), bought the Excel centre for a $1billion.

“Apparently,” said the cabby, “they do a lot of, you know, ‘God conferences’, evangelical like, yeah?”

“Oh?” I enquired—shocked because I’d been greeted by some charismatics in red teeshirts on my first visit to FOWA, and they’d been there every morning, greeting folks entering the conference and—strangely—leaving the toilets.

“Yeah, there’s a lot of money in it over in the States. Folks that run ‘em want to come to England, and get some British God-money, I guess. Problem for them is: there aren’t any places big enough! So, they buy it, and are building a bigger place. Lots of dosh!”

What?

I had cynically been thinking to myself that the red-tee-shirted cohort of pentecostals greeting me by name (no, nothing spiritual—we all wear name-tags!) had been selected for their good looks and ethnic diversity. But I thought I was being cynical.

But here’s the thing: I overheard maybe ten conversations between geeks (It was a web conference, after all) about the “religious people/ god-folk/ Christian nutters”, and my heart sank. One or two said that a cheery greeting first thing in the morning is quite nice. Most, however, thought they were over-bearing, weird, or strangely sexy.

The thing that gets me here, is that the public image of Jesus’ followers is horribly maimed here. Wasn’t it Jesus who through money-changers out of the temple? Wasn’t it Jesus who went to societies low-lifes and changed their worlds? You think he would have scored well with a PR agency? Think he had a Marketing team or cheer-leaders?

“The only thing spiritual they [Arab investors in the venue] get is this:” and the cabby rubbed his fingers in the international symbol of money.

What should I say to this? I’m lost for words…

Unfavourable rail network numbers

// April 29th, 2008 // Comments // travel

I have come to the conclusion, after being without a car for five years, that travel by rail in the UK is just about the most infuriating process you can undertake on a regular basis. Every day, one of my 6 train journeys to or from work will fail in some way or other. Either there will be a delay, a last-minute platform change, a cancellation… something goes wrong.

Looking at the complexity of the rail network, it’s not surprising that it doesn’t always run exactly right. However, since I can pretty much guarantee that my train journey to work or back will be compromised in some way, I’m starting to wonder. There is no ‘normal’ service, you see. It’s a myth that the trains usually make it on time. Perhaps it is more than 50% of each leg of any given journey… maybe. But at some stage in every multi-sectioned journey I’ve had, some aspect of the trip fails.

II’m beginning to wonder what could possibly fix this situation. I’m told daily by an electronic voice: “I’m extremely sorry for the severe delay to this service.” I wonder what rail executives hope to achieve by creating an automated apology system? It’s not exactly like it’s a good value for money trade-off either. Train travel is bloody expensive, and it doesn’t work the majority of the time.

Maybe executive heads need to roll, maybe a huge infusion of public money needs injecting into the system, maybe… they need a change of perspective. If they measure every individual step in a day, they might come to the conclusion that they’ve failed n number of times. n=number of delayed, cancelled or otherwise compromised phases of transit between any two given points on the network. However, if X represents the full journey of an individual in a sampled population, and X is made up of a number of phases, I wonder how many X’s would end up compromised in some way?

Sample 100 complete journeys across the entire network. Each journey is an individual X (complete from point A to B with each stop and change included). Each completed leg of any given X could be designated y, and any compromised legs (including additionally-accrued attrition from one failed leg compromising another) could be w’s.

There are two numbers I’d be interested in seeing:

• The ratio of y’s to w’s across the sampled population’s collective X’s
• The number of X’s which contain at least one w

I reckon and predict, that these numbers would reflect very unfavorably on the network’s ability to run itself.

What’s the fix?

No idea… though I do worry no one in the rail network has any clue either.