How the Met Office is handling a deluge of data

'Lazy’ open source tools are helping the UK weather forecasting service open up unprecedented volumes of data to all.

IT industry commentators often quote mind-boggling figures to illustrate the huge volumes of data organisations will have to store and process in future. But if you want a more accurate forecast of the challenges you’re likely to face when data gets really big, ask the Met Office.

Home to some of the world’s largest supercomputers, the organisation today holds around 60PB (petabytes) of data, actively processing around a petabyte a day. Its data archive is increasing by around 1.4PB a week and by 2020 will contain more than 300PB.