How Facebook Ships Code

by yeeguy

I’m fascinated by the way Facebook operates.  It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried).  These are notes gathered from talking with many friends at Facebook about how the company develops and releases software.

Seems like others are also interested in Facebook…   The company’s developer-driven culture is coming under greater public scrutiny and other companies are grappling with if/how to implement developer-driven culture.   The company is pretty secretive about its internal processes, though.  Facebook’s Engineering team releases public Notes on new features and some internal systems, but these are mostly “what” kinds of articles, not “how”…  So it’s not easy for outsiders to see how Facebook is able to innovate and optimize their service so much more effectively than other companies.  In my own attempt as an outsider to understand more about how Facebook operates, I assembled these observations over a period of months.  Out of respect for the privacy of my sources, I’ve removed all names and mention of specific features/products.  And I’ve also waited for over six months to publish these notes, so they’re surely a bit out-of-date.   I hope that releasing these notes will help shed some light on how Facebook has managed to push decision-making “down” in its organization without descending into chaos…  It’s hard to argue with Facebook’s results or the coherence of Facebook’s product offerings.  I think and hope that many consumer internet companies can learn from Facebook’s example.

HUGE thanks to the many folks who helped put together this view inside of Facebook.   Thanks are also due to folks likeepriest and fryfrog who have written up corrections and edits.


  • as of June 2010, the company has nearly 2000 employees, up from roughly 1100 employees 10 months ago.  Nearly doubling staff in under a year!
  • the two largest teams are Engineering and Ops, with roughly 400-500 team members each.  Between the two they make up about 50% of the company.
  • product manager to engineer ratio is roughly 1-to-7 or 1-to-10
  • all engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs and listening to lectures given by more senior/tenured engineers.  estimate 10% of each boot camp’s trainee class don’t make it and are counseled out of the organization.
  • after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data)
  • [EDIT thx fryfrog“There are also very good safe guards in place to prevent anyone at the company from doing the horrible sorts of things you can imagine people have the power to do being on the inside. If you have to “become” someone who is asking for support, this is logged along with a reason and closely reviewed. Straying here is not tolerated, period.”
  • any engineer can modify any part of FB’s code base and check-in at-will
  • very engineering driven culture.  ”product managers are essentially useless here.” is a quote from an engineer.  engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime.  [EDITORIAL] The author of this blog post is a product manager, so this sentiment really caught my attention.  As you’ll see in the rest of these notes, though, it’s apparent that Facebook’s culture has really embraced product management practices so it’s not as though the role of product management is somehow ignored or omitted.  Rather, the culture of the company seems to be set so that *everyone* feels responsibility for the product.
  • during monthly cross-team meetings, the engineers are the ones who present progress reports.  product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that “product spoke too much at the last meeting.”  they really want engineers to publicly own products and be the main point of contact for the things they built.
  • resourcing for projects is purely voluntary.
    • a PM lobbies group of engineers, tries to get them excited about their ideas.
    • Engineers decide which ones sound interesting to work on.
    • Engineer talks to their manager, says “I’d like to work on these 5 things this week.”
    • Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first.
    • Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between.  If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on.  Same for Architect help.  But in general, expectation is that engineers will handle everything they need themselves.
  • arguments about whether or not a feature idea is worth doing or not generally get resolved by just spending a week implementing it and then testing it on a sample of users, e.g., 1% of Nevada users.
  • engineers generally want to work on infrastructure, scalability and “hard problems” — that’s where all the prestige is.  can be hard to get engineers excited about working on front-end projects and user interfaces.  this is the opposite of what you find in some consumer businesses where everyone wants to work on stuff that customers touch so you can point to a particular user experience and say “I built that.”  At facebook, the back-end stuff like news feed algorithms, ad-targeting algorithms, memcache optimizations, etc. are the juicy projects that engineers want.
  • commits that affect certain high-priority features (e.g., news feed) get code reviewed before merge. News Feed is important enough that Zuckerberg reviews any changes to it, but that’s an exceptional case.
  • [CORRECTION — thx epriest“There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn’t look at every change personally.”
  • [CORRECTION thx fryfrog“All changes are reviewed by at least one person, and the system is easy for anyone else to look at and review your code even if you don’t invite them to. It would take intentionally malicious behavior to get un-reviewed code in.”
  • no QA at all, zero.  engineers responsible for testing, bug fixes, and post-launch maintenance of their own work.  there are some unit-testing and integration-testing frameworks available, but only sporadically used.
  • [CORRECTION thx fryfrog“I would also add that we do have QA, just not an official QA group. Every employee at an office or connected via VPN is using a version of the site that includes all the changes that are next in line to go out. This version is updated frequently and is usually 1-12 hours ahead of what the world sees. All employees are strongly encouraged to report any bugs they see and these are very quickly actioned upon.”
  • re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code.  it’s just that they don’t have an incentive to do so at most companies.  when there’s a QA department, it’s easy to just throw it over to them to find the errors.”  [EDIT: please note that this was subjective opinion, I chose to include it in this post because of the stark contrast that this draws with standard development practice at other companies]
  • [CORRECTION thx epriest“We have automated testing, including “push-blocking” tests which must pass before the release goes out. We absolutely do not believe “most engineers are capable of writing bug-free code”, much less that this is a reasonable notion to base a business upon.”
  • re: surprise at lack of PM influence/control — product managers have a lot of independence and freedom.  The key to being influential is to have really good relationships with engineering managers.  Need to be technical enough not to suggest stupid ideas.  Aside from that, there’s no need to ask for any permission or pass any reviews when establishing roadmaps/backlogs.  There are relatively few PMs, but they all feel like they have responsibility for a really important and personally-interesting area of the company.
  • by default all code commits get packaged into weekly releases (tuesdays)
  • with extra effort, changes can go out same day
  • tuesday code releases require all engineers who committed code in that week’s release candidate to be on-site
  • engineers must be present in a specific IRC channel for “roll call” before the release begins or else suffer a public “shaming”
  • ops team runs code releases by gradually rolling code out
    • facebook has around 60,000 servers
    • there are 9 concentric levels for rolling out new code
    • [CORRECTION thx epriest“The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.”
    • the smallest level is only 6 servers
    • e.g., new tuesday release is rolled out to 6 servers (level 1), ops team then observes those 6 servers and make sure that they are behaving correctly before rolling forward to the next level.
    • if a release is causing any issues (e.g., throwing errors, etc.) then push is halted.  the engineer who committed the offending changeset is paged to fix the problem.  and then the release starts over again at level 1.
    • so a release may go thru levels repeatedly:  1-2-3-fix. back to 1. 1-2-3-4-5-fix.  back to 1.  1-2-3-4-5-6-7-8-9.
  • ops team is really well-trained, well-respected, and very business-aware.  their server metrics go beyond the usual error logs, load & memory utilization stats — also include user behavior.  E.g., if a new release changes the percentage of users who engage with Facebook features, the ops team will see that in their metrics and may stop a release for that reason so they can investigate.
  • during the release process, ops team uses an IRC-based paging system that can ping individual engineers via Facebook, email, IRC, IM, and SMS if needed to get their attention.  not responding to ops team results in public shaming.
  • once code has rolled out to level 9 and is stable, then done with weekly push.
  • if a feature doesn’t get coded in time for a particular weekly push, it’s not that big a deal (unless there are hard external dependencies) — features will just generally get shipped whenever they’re completed.
  • getting svn-blamed, publicly shamed, or slipping projects too often will result in an engineer getting fired.  ”it’s a very high performance culture”.  people that aren’t productive or aren’t super talented really stick out.  Managers will literally take poor performers aside within 6 months of hiring and say “this just isn’t working out, you’re not a good culture fit”.  this actually applies at every level of the company, even C-level and VP-level hires have been quickly dismissed if they aren’t super productive.
  • [CORRECTION, thx epriest]  “People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren’t around to support them in case something goes wrong (and haven’t found someone to cover for you).”
  • [CORRECTION, thx epriest“Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature.”
  • [CORRECTION, thx fryfrog] “I also don’t know of anyone who has been fired for making mistakes like are mentioned in the article. I know of people who have inadvertently taken down the site. They work hard to fix what ever caused the problem and everyone learns from it. The public shaming is far more effective than fear of being fired, in my opinion.”

It’ll be super interesting to see how Facebook’s development culture evolves over time — and especially to see if the culture can continue scaling as the company grows into the thousands-of-employees.

What do you think?  Would “developer-driven culture” work at your company?

Permalink |

203 thoughts on “How Facebook Ships Code

  1. Pingback: link vao m88

  2. Pingback: papas al

  3. Pingback: download free

  4. Pingback: apps

  5. Pingback: Hindi Sex Story

  6. Pingback: Skrota bilen

  7. Pingback: trump

  8. Pingback: insulation

  9. Pingback: VA Home Loans

  10. Pingback: bad credit home loans

  11. Pingback: mark curry sol

  12. Pingback: samurai tour

  13. Pingback: hernia mesh lawsuit

  14. Pingback: cloud visionary

  15. Pingback: motu and patlu

  16. Pingback: hampton bay light fixtures

  17. Pingback: best whatsapp status for hindi

  18. Pingback: boldabol

  19. Pingback: hire an attorney

  20. Pingback: w88live

  21. Pingback:

  22. Pingback: UK Chat Rooms

  23. Pingback:

  24. Pingback: economics tuition

  25. Pingback: published here

  26. Pingback: Skrotbilspremie

  27. Pingback: trouver des gens sur telephone

  28. Pingback: mca

  29. Pingback: Best Best Online Tamil News in the World

  30. Pingback: Best Best Online News

  31. Pingback: Best Best Online News in the World

  32. Pingback: Best Best Online News in the World

  33. Pingback: Best Best Online News in the World

  34. Pingback: Best Best Online News in the World

  35. Pingback: hermes ceinture

  36. Pingback: barbour for sale

  37. Pingback: cheap marmot jackets

  38. Pingback: mens moncler jackets on sale

  39. Pingback: barbour outlet online

  40. Pingback: coach online outlet

  41. Pingback: north face factory outlet

  42. Pingback: hermes sydney store

  43. Pingback: ugg adirondack boot sale

  44. Pingback: hermes belt on sale

  45. Pingback: parajumpers jackets outlet

  46. Pingback: ugg adirondack boot sale

  47. Pingback: Michael Kors Handbags Outlet

  48. Pingback: mammut jacket sale

  49. Pingback: cheap louis vuitton sale

  50. Pingback: hermes montres homme

  51. Pingback: discount michael kors

  52. Pingback: barbour outlet online

  53. Pingback: hermes store online

  54. Pingback: belstaff sale mens

  55. Pingback: mens hermes belts for sale

  56. Pingback: pierre hardy espadrilles

  57. Pingback: hermes bracelet

  58. Pingback: chanel outlet store

  59. Pingback: sac kelly hermes prix

  60. Pingback: belstaff online

  61. Pingback: roberto cavalli sample sale

  62. Pingback: Victoria's Secret on sale

  63. Pingback: arcteryx online store

  64. Pingback: pierre hardy canada

  65. Pingback: ray bab

  66. Pingback: ray ban clubmaster prezzo

  67. Pingback: scannable fake id uk

  68. Pingback: parajumpers jackets outlet

  69. Pingback: fake id vs real

  70. Pingback: fake id proof of age

  71. Pingback: fake id picture

  72. Pingback: arcteryx therme sale

  73. Pingback: sergio shoes

  74. Pingback: occhiali da sole ray ban

  75. Pingback: arcteryx corporate sales

  76. Pingback: fake id cheap

  77. Pingback: occhiali da sole oakley prezzi

  78. Pingback: fake id guide

  79. Pingback: wisconsin fake id

  80. Pingback: patagonia student discount

  81. Pingback: fast fake id

  82. Pingback: occhiali ray ban in offerta

  83. Pingback: ontario fake id

  84. Pingback: fake id provisional

  85. Pingback: Rene Caovilla outlet

  86. Pingback: saut hermes paris

  87. Pingback: north face store online

  88. Pingback: us fake id

  89. Pingback: using fake id

  90. Pingback: cheap arcteryx

  91. Pingback: fake id drivers license

  92. Pingback: us fake id

  93. Pingback: fake id us

  94. Pingback: Callaway Golf outlet

  95. Pingback: fake id felony

  96. Pingback: drivers license fake

  97. Pingback: Barbour International Jackets

  98. Pingback: fake id photo

  99. Pingback: parajumpers new york

  100. Pingback: prada buy

  101. Pingback: charges for fake id

  102. Pingback: sunglasses oakley sale

  103. Pingback: coach factory online

  104. Pingback: Stuart Weitzman outlet online

  105. Pingback: make fake id

  106. Pingback: pennsylvania fake id

  107. Pingback: colorado fake id

  108. Pingback: fake australian id

  109. Pingback: fake id photo

  110. Pingback: cole haan shoes outlet

  111. Pingback: how to use a fake id

  112. Pingback: louis vuitton online bags

  113. Pingback: toms shoes outlet

  114. Pingback: fake id cost

  115. Pingback: iowa fake id

  116. Pingback: fake id machine

  117. Pingback: patagonia outlet

  118. Pingback: do fake id reviews

  119. Pingback: fake california id

  120. Pingback: fake school id

  121. Pingback: arcteryx stingray jacket sale

  122. Pingback: belstaff gilet sale

  123. Pingback: balmain jeans womens

  124. Pingback: cheap roger vivier

  125. Pingback: fake NewHampshire license

  126. Pingback: coach outlet online coupon

  127. Pingback: sneakers golden goose

  128. Pingback: fake caller id

  129. Pingback: fake id review

  130. Pingback: ray ban outlet

  131. Pingback: cheap rayban

  132. Pingback: buy cheap oakleys

  133. Pingback: cheap ray ban sunglasses outlet

  134. Pingback: ray ban sunglasses outlet

  135. Pingback: cheap ray bands

  136. Pingback: ray ban cheap

  137. Pingback: oklahoma fake id

  138. Pingback: fake id oregon

  139. Pingback: reddit fake id

  140. Pingback: cheap fake id

  141. Pingback: illinois fake id

  142. Pingback: fake id for sale

  143. Pingback: where can you buy a fake id

  144. Pingback: fake school id

  145. Pingback: Tennessee fake id template

  146. Pingback: get a fake id

  147. Pingback: pa fake id

  148. Pingback: reddit fake id

  149. Pingback: what do you need to get a florida id

  150. Pingback: fake id places nyc

  151. Pingback: id florida

  152. Pingback: Maryland fake id laws

  153. Pingback: where do you get fake ids

  154. Pingback: how can you tell if an id is fake

  155. Pingback: fake id Columbus

  156. Pingback: order fake ids

  157. Pingback: fake id proof of age

  158. Pingback: Philadelphia fake id

  159. Pingback: fake id blacklight

  160. Pingback: fake pennsylvania id

  161. Pingback: Virginia fake id laws

  162. Pingback: fake ids toronto

  163. Pingback: how to make a fake Pennsylvania driver's license

  164. Pingback: fake id fraud

  165. Pingback: picture of michigan drivers license

  166. Pingback: easiest states to fake id

  167. Pingback: fake Arkansas driver's license

  168. Pingback: how to make a fake Delaware driver's license

  169. Pingback: buy NewYork fake id

  170. Pingback: scannable

  171. Pingback: good fake id sites

  172. Pingback: fake Virginia id generator

  173. Pingback: fake id houston

  174. Pingback: fake id card maker

  175. Pingback: where to use fake id

  176. Pingback: ontario fake id

  177. Pingback: the best fake id website

  178. Pingback: fake irish id

  179. Pingback: getting a florida id

  180. Pingback: buy fake id online

  181. Pingback: Idaho fake id template

  182. Pingback: fake florida drivers license

  183. Pingback: fake Colorado id

  184. Pingback: how to make fake id australia

  185. Pingback: real fake id reviews

  186. Pingback: fake id washington

  187. Pingback: fake id chicago

  188. Pingback: fake drivers permit

  189. Pingback: fake id orlando

  190. Pingback: boutique hermes bordeaux

  191. Pingback: Architecture Energy

  192. Pingback: International Paediatric Haematology and Oncology Update Meeting

  193. Pingback: PR Letting Solutions

  194. Pingback: hermes vaudreuil recrutement

  195. Pingback: golden goose paris

  196. Pingback: golden goose homme pas cher

  197. Pingback: ecole hermes marseille

  198. Pingback: bracelet hermes femme

  199. Pingback: golden goose homme soldes

  200. Pingback: golden goose 2017

  201. Pingback: hermes conf

  202. Pingback: le grand saut hermes

  203. Pingback: golden goose paris

Leave a Reply