August'24: Kamaelia is in maintenance mode and will recieve periodic updates, about twice a year, primarily targeted around Python 3 and ecosystem compatibility. PRs are always welcome. Latest Release: 1.14.32 (2024/3/24)

Deliver The Archive?

The BBC audio archive comprises of 500,000 hours of audio. At bit rates for an iPod this is a mere 27TB of storage. In terms of current technology this would fit on commodity disk stores which could sit under a desk. In terms of solid state storage - you'd need 27,000 MMC cards. In practice that means 1 cubic foot of storage. Using video bit rates used for the Olympics you're talking either 3 desks for disk based storage, or 1 shelf full for solid state.

> gutenberg project

Clearly these bit rates are not suitable for archival and would be inappropriate for television production. The idea that even the entire BBC archive could fit in most people's homes today is a startling one.

If the amount of the archive the BBC chose to make available was as much as 10%, then it becomes realistic for the audience to download now, choose later - even on a laptop - if storage continues to increase as projected.

In this world it makes more sense to deliver the content available en masse to the entire audience than it does to pick and choose. Obviously there are technical hurdles to jump over in achieving this, but conversely having the content almost everywhere massively simplifies this distribution. We allow the network and users to populate the network. It seems inefficient to force the audience to return to the BBC for content when they will be able to locally store all the content we make available.

At this point in time the question then becomes not "how do we serve millions of concurrent streams", but rather "how do we ensure that we populate the millions of storage devices out there?"

Challenge: How do we deliver a BBC repository, dozens of Terabytes in size, to every home in the UK, regularly refreshed, in a way that copes with disk failures and data loss.