Platform
- Twice - custom web caching system. (http://code.google.com/p/twicecache/)
- XFS - file system.
- HAProxy - software load balancing.
- The LVS stack and ldirectord - high availability.
- Ruby on Rails - application server
- Nginx - web server.
- PostgreSQL - database used for user and other meta data.
- MongoDB - used for their internal analytics tools.
- MemcachedDB - used for handling high write data like view counters.
- Syslog-ng - logging service.
- RabitMQ - used for job system.
- Puppet - used to build servers.
- Git - used for source code control.
- Wowza - Flash/H.264 video server, plus lots of custome modules written in Java.
- Usher - custom business logic server for playing video streams.
- S3 - small image storage.
The Stats
- 4 datacenters spread through out the country.
- At any given time there's close to 2,000 incoming streams.
- 30 hours per minute of video is added each day.
- 30 million unique visitors a month.
- Average live bandwidth is about 45 gigabits per second. Daily peak bandwidth at about 110 Gbps. Largest spike has been 500 Gbps.
- Approximately 200 video servers, based on commodity hardware, each capable of sending 1Gbps of video. Smaller than most CDNs yet larger than most video websites.
- About 100TB of archival storage is saved per week.
- The complete video path can't have more than 250 msecs of latency before viewers start losing the ability to converse and interact in real-time.
Well, I didn't use Justin.tv myself, so I can't tell you how well does this work; however, it looks like they took all the simple bits and pieces and built a decent platform from them.
