Methodology: How The 50 Best Restaurants List Came To Be

See that little dancing picture up there? That’s how the list of Philadelphia’s 50 Best Restaurants came to be. That’s one wall of Foobooz World HQ, covered completely with 170-odd Post-It Notes, representing every single restaurant that was in contention for a spot in the 2012 rankings. That’s what we lived with for roughly three months–a constantly shifting and evolving representation of Philadelphia’s restaurant scene to be stared at, debated, argued over, torn down, rebuilt and obsessed over until me and Art and the rest of this year’s 50 Best Restaurants crew had gone all A Beautiful Mind and saw the thing in our sleep.

How did Stateside end up at #1? Why didn’t Fette Sau or Rittenhouse Tavern make the list at all? And what were we thinking with Marigold Kitchen? There are answers to all of those questions, and it all began back in August with a deceptively complicated question:

What are your favorite restaurants in Philadelphia?

That was the question that I put to the staff of Philadelphia magazine at the start of this. That was the question that I asked friends and neighbors, cooks and bartenders, relatives, foodie scenesters and strangers in the elevator. Not “What are the best restaurants in Philadelphia,” but what are your favorites–where do you go to celebrate, to hang out, to seek comfort or to show off?

To ask people about the “best” restaurants in any city is to get back answers freighted with too much noise: They give you the answers that they think will make them sound sophisticated and well-schooled, that will show the city in the best light, that represent a cross-section of perceived knowledge and historical fantasy. For Philadelphia, it also means getting back a list which I think anyone reading this blog could probably rattle off in a single breath. “Well of course it would be VetriLeBecFinFountainLacroixBlackfishZahav.” Doesn’t matter that they haven’t been to Le Bec since the changeover, or that they’ve only ever heard about dinners at Fountain. Doesn’t matter that chef changes and concept changes and overhauls of rooms and menus have occurred that have fundamentally altered their approach to the changing tastes of the fickle public. There are white tablecloths and white truffles and gold cards and epic wine lists and folks in the dining room who look like they’re auditioning for the part of Mr. Monopoly, and that’s all that matters. When you ask people about the best, they almost always go high. It’s a gut reflex, and not always wrong (all of those places are great restaurants, every one of them is on the list), but it isn’t always totally right either.

But ask about favorite restaurants, and you get a very different list. A crazy list. A wildly divergent list that speaks of need and desire as much as it does the cutting edge of newness, and also a long list. In 2012, it was a list of 171 different restaurants, covering everything from the Austrian Village and Penang to Le Bec and Vetri. That was where we started five months ago. In August, anything was possible. Nothing was out of bounds.

The list came down slowly as The Wall became a named thing and a permanent feature in our office. We knocked out restaurants that had closed (in some cases years ago) and restaurants that were quirky, one-off faves for reasons that had nothing to do with any reasonable expression of Best-ness (I love Penang, for example, because I fell in love at Penang and because it serves the best roti canai in America, but I don’t honestly believe it is one of the 50 best restaurants in the city). We made a special section called “The Dead Zone” for places that we knew were never going to make the cut (places like Serafina, Naan, Geno’s, Nan Zhou, Tapestry and Square Peg), then began winnowing the list down to 150, 120, 110…

When we got to 101, we hit a kind of wall. 101 was the point where we could no longer simply dismiss restaurants out of hand, and knew that we had to start eating.

And eating. And eating more. 101 of Philadelphia’s best restaurants got anywhere between 1 and 7 visits between the beginning of September and the end of November, and this is the first point at which I get to laugh a little at all the furious readers lighting up the comments section and accusing us of everything from rank idiocy to payola. To the most apoplectic among them, I have a simple retort: We ate more than you. We ate A LOT more than you. And while that doesn’t make us automatically right, it makes us quite a bit more right than you who eat merely recreationally. Are you upset that your favorite restaurant didn’t make the list? Don’t worry. We ate there. And we had our reasons. Pissed that some place was ranked higher than you think it ought to be? I promise you that there was good cause.

Take Marigold Kitchen, for example. When we began, Marigold was among the 170. It survived all the way to 101, but was really lingering on the cusp of the Dead Zone because, like you, we all thought, “Really? But no one has talked about that place in forever…”

But then I got an email from a trusted eater saying that we had to send someone back there immediately. That the kitchen, which had always been good, had suddenly become remarkable. After that, I was stopped in the hall by someone else who said that they’d just had the best meal of their year and had to tell me about it immediately. They, too, had just come back from Marigold.

I sent a scout. She had a phenomenal meal and Marigold rose into an early version of the Top 50, topping out somewhere around the 20’s. We looked over Trey Popp’s most recent revisit, and adjusted it into the high teens. In October, I sent Trey back in again for another meal and he came back singing the praises of chef Robert Halpern, saying that he, too, had had one of the best meals of his year there. I had forms that all of our commando eaters were supposed to fill out for every meal they ate, and when Trey’s came back to be, Marigold Kitchen outranked both Le Bec and Vetri. Marigold moved to number 7 or 8 and hung there for a very long time while the list took shape around it.

Hundreds of dinners, dozens of eaters, thousands of dollars, hours of arguing… Rittenhouse Tavern was an early surprise victim of the process–not one bad meal there, but three, balanced against one pretty good one that I had personally. And then one of the staff came back furious because she’d had not just a bad meal, but the worst meal while under the care of the Tavern’s staff. So bad that she couldn’t stop listing the things that had gone wrong. It became too much of a trend to ignore. Ela was inconsistent at best. At worst, there was a sense that a plateau of talent had been reached, and that the plateau just wasn’t that high. Compare that to Will or Vernick or Fork which scored as highly as they did not solely because successive meals there were all excellent, but because there was a consistent sense that each of them was only getting better.

Speaking of Will… It was decided fairly early on that the opening of Will would be our cut-off for eligible restaurants. Anyone who opened after Will would have to wait for the next time we do one of these projects. Curious as to why Fette Sau didn’t make the 2012 list? That’s the reason. The only reason. And those were tough guns to stick to.

Back to Marigold. By November, the list had begun to take shape, but the top 15 or so were still very much up in the air. A hardcore few of us were still eating, making 2nd or 3rd or 5th visits to places that were giving us trouble. At this point, though, it was mostly about talking. And talking. And talking and talking and talking and talking. We had meetings. We discussed positions over morning coffee and afternoon whiskey and argued about the failings of the pure math approach–at which point did basing the list on numbers alone give a false result, and how much weight ought to be given to the impassioned defense of those who had actually eaten there.

Marigold climbed the notches. Vetri ended up a slot below Amis. Zama clawed its way onto the list on the strength of two absolutely phenomenal meals and Morimoto got ditched on the blandness of two wholly mediocre ones. Stateside bounced from #10 to #1 over the course of two weeks but, really, just one afternoon when I, myself, had to face down the question of where I would rather eat, Stateside or Sbraga, Stateside or Vetri? I love Stateside because I believe it represents the new vanguard of American cuisine. More than that, it seems to be pointing the direction. I love it because I have never had a bad meal there, or even a bad plate. I love it because the underlying notion of the place–that this is Pennsylvania’s cuisine–is not narrow, but rather so inclusive of everything one might eat in Pennsylvania, from Korean comfort food to some of the best charcuterie anywhere.

The list was still changing right up until the very last possible moment. Then it continued changing even as we were assembling the issue for publication. There was never a moment when we were sure that we had everything right because anyone who says that their list of the best anything anywhere is the absolute ultimate last word forever and ever is just an idiot and a fool, but I am confident that this list, as it stands, says something important about this moment in the history of Philadelphia’s restaurant scene.

It says that things are changing. That the future is unclear, but potentially awesome. It says that the old guard–the white tablecloths, the wine lists, the jacket-and-tie temples of culturally staid Gastronomy-with-a-capital-G–are waning in influence, but struggling mightily to gain it back and, in some cases (Fountain) succeeding. And it says that, in the meantime, there has never been a better time in Philadelphia to be looking for a place to eat.

Philadelphia’s 50  Best Restaurants [Official]

Be respectful of our online community and contribute to an engaging conversation. We reserve the right to ban impersonators and remove comments that contain personal attacks, threats, or profanity, or are flat-out offensive. By posting here, you are permitting Philadelphia magazine and Metro Corp. to edit and republish your comment in all media.

  • alex

    so MAYBE YOU SHOULD CALL THE LIST “OUR 50 FAVORITE RESTAURANTS” INSTEAD OF “50 BEST RESTAURANTS”. Just a fucking thought.

  • Actuary

    Thank you for opening the window into your selection process.
    Unfortunately all it does is highlight sampling and statistical flaws in the process. Arbitrary rules like 1 person has a bad meal at a restaurant thus deleting the restaurant only relegates al attempt at logical selection to silly emotional conclusions. Also simply saying some places were not going to make the cut without further explanation as to why is methodically bizzarre if not tinged with politics or bias. It ultimately calls your conclusive list highly into question. Stateside is a good restaurant. George Sabatino is a smart talented chef. If your samplers conclude that Stateside is the #1 restaurant in Philadelphia they are out of their minds. Even the chef of Stateside can not publicly stand by the statement that the food and/or service at stateside is the best in Philadelphia. that is what makes this list a joke. I know it has been discussed but a few things are not subjective. Federal Donuts is not even a restaurant. Zeppoli is in another state for christ’s sake. If you cant pick the top restaurants in a city without picking restaurants in another state other than the one that city is located in then gord help you. This methodology is flawed and the conclusions are clearly silly as is evidenced by the overwhelmingly negative response.

  • They call me Irony

    I agree with Alex’s last statement minus the sailor slang.

    Executing a competition is very difficult as I know from past experience. There are so many factors involved that you want to stay away from. The biggest problem that I have found when creating a competition was keeping the “favoritism” at bay while judging. A people’s choice award can also be skewed to how cool you think the person is rather than what is being served in front of you. I would personally keep the rankings to limited number- say 25. Just as the NCAA would rank College Football Division I. I would only request that your judges have a educated and defined palate and to have fun with the competition.
    As for the statistical analysis from comment 2- you don’t know what the heck you are talking about, move along…

    Cheers!

  • captain obvious

    So rittenhouse tavern got the axe because some intern had a bad meal there once?? and ela the axe because Eater has already given the love to Chiconski?? very credible list guys…but its all good, I’m surprised Hop sing didn’t make the cut

  • Haterade

    Then don’t buy the magazine or come to the site. Serious question: Why do you come here if you hate it and them so much?

  • rory

    First, fuck you for claiming we have no right to be critical. Seriously, GTFOH.

    Secondly, fuck you for putting nan zhou in the dead zone. Seriously, GTFOH again (though the move has weakened the broth, imo).

    Thirdly, the method you used has a serious problem: its you, your buddies, and your colleagues. This, then, explains the lack of coverage of entire areas and types of cuisines.

    Fourth, again, this post runs exactly counter to Ashley’s post. You two should talk one of these days.

    Fifth, this list is talking about a future change that’s been occurring and identified for at least five years or more. Yawn at your claptrap about white tableclothes and wine lists. The mid 2000s called, they want their excitement about bare tables and lack of pretension back.

    Finally, another serious problem with your methodology is that you didn’t define what makes a great restaurant before the process. All good research starts with a well defined research question. This fails that basic premise. That’s why you ended up going with a bizarre mish-mash of unequal tastings, impassioned defenses, and btw, Stateside’s premise is AMERICAN made, not PA cuisine (hence GA sourced grits and a bourbon theme and NO PA CHEESES ON THEIR CHEESE MENU, broccolini instead of rabe, and oysters). at least get the facts about your #1 restaurant right. I’m not even hating on stateside (yet…), as I enjoyed my meals there. My friends, on the other hand, didn’t…so whatever. they had two off nights there, I guess. They don’t eat out enough to judge, tho, right? lol.

    Also, I don’t know how the #1 restaurant can have a high-fallutin’ cocktail program as one of its selling points and make unbalanced cocktails so routinely (also, they use bad ice. and that matters). Then again, i already knew your bar list was bad, so i’m not surprised.

  • Mike

    It seems that opening up your thought process to how restaurants were selected is admitting to a very flawed list. It comes to light that the panelists did not select the best restaurants based on food, service, and atmosphere (and seeing as Fed Donuts made it, establishments did not even need to be a restaurant at all), but they selected places that they believe (from a very biased view) are the most-fun or their favorite restaurants. This is not the best restaurants in our town, but more of a FooBooz guide to eating in Philadelphia (which is okay, but definitely not the best culinary experiences nor “The 50 Best”).

  • Anthony

    Or

    “Our Top 50 Favorite Philadelphia Restaurants”

    FTFY

  • Alimentarian

    Why don’t you just admit that you came up with this list by having monkeys throw darts at the restaurant names.

  • Lindsay

    I cant believe how upset people are getting over all of this….its not that big of a deal! No one is forcing anyone to live and die by this list. Its a publication’s opinion…and I think its pretty comprehensive list of some of the city’s best places to eat. Every single one? Maybe not. A couple weird ones? Probably. But It doesn’t mean you have to go eat there, and it doesn’t mean you cant eat somewhere that didn’t make the list.

  • bobsbigboy

    I wonder how many of these commenters actually work in all of the hard working restaurants on the list?

  • chooseyourweapon

    i wonder if diners in other cities think they are as brilliant as philadelphians?

  • Nina

    I think this is awesome! There is not science behind picking a best or favorite, or whatever you want to call it restaurant list. If you are not happy with the list, patronize the restaurants you think should be on there. I love seeing a shake up to the standard rankings, Philadelphia is full of amazing, interesting and creative restaurants, be happy that you have 50 restaurants worth talking about!

  • barryg

    Why does Nan Zhou make the dead zone but not FedNuts? The answer to that will add a lot of lucidity to the process. Also I wonder why Naan and Tapestry were put in the dead zone rather than removed along with other closed restaurants?

  • hiki

    I think thou does protest too much, trying to dispove advertising has anything to do with Philadelphia Magazines “Best” of Lists. Everyone in Philadelphia, who works in any establishment that has the potential to be put on one of their lists, knows the truth about those lists.

  • Charlie

    Just pulled out your list from the 2006 version. Not that any of you were there back then but, 60% of the restaurants are closed & re-opened 2-3 times over. I guess this list is moot. Does it really matter ? Will Stateside put up a special marker on their window promoting your listing ? Doubt it.

    Did you have someone outside of your offices give feedback ? Maybe not just hear someone walking by you in the hallway. Agree with others that this list can be viewed as a “favorite” list and not the best. By the way – Federal Donuts is highly overrated – neither a good restaurant? nor good donuts.

    The list should be titled our 50 favorite restaurants right now – because by early summer there will be an opening you’ll promote 20 times on this site and fall in love before 2013 is over and that place will top next year’s list.

    Happy New Year’s and Happy Eating !! Excited to live and eat in Philly!

  • jay

    rory, correct me if i’m wrong, but didn’t you used to vote in the top 50 bars? And you used to defend it. Now you don’t, and you attack it.

    I’m not saying your attacks are without merit, but I am saying they may be biased. Did you have a falling out with Art?

  • Frank

    Key quote: “There was never a moment when we were sure that we had everything right because anyone who says that their list of the best anything anywhere is the absolute ultimate last word forever and ever is just an idiot and a fool.”

    If your ignorance prevents you from realizing that any list, regardless of title, is just a conversation starter and isn’t “definitive” unless you perceive it to be definitive, that’s your problem. Shut the hell up and eat wherever you want already.

  • CM Drunk

    Using your own rules and using Will as a cutoff, Fork is ineligible as their new chef arrived well after Will opened establishing the essentially new restaurant.

  • Simplified

    The methodology can be boiled down to this “Art’s a douchebag and this is the list the douchebag came up with”

  • rory

    This is jasons list. I couldnt pick art out in a crowd, nor he me.

    I hate philly mags coverage. I liked arts foobooz far more. I contributed to that, not to this. I hate this because of all the reasons ashley wrote.

  • redonk

    It strikes me, having waited on Frank Bruni 3x’s (3stars)& LeBan 3x’s (also 3bells), that none of the critics that I hear from these lower tiered blogs & mags have absolutely no culinary degrees amongst them. Why is that? Why do we continue to jump through hoops for them? Why do we allow these undereducated, lackies to tell us who is “leading the way”? You send your idiots to our restaurants & we usually end up comping either all or part of your bill as a courtesy for you to run willy-nilly at the mouth as if you were prophets.
    The 50 list is a Loud Emample that although Philly can definitely hang with the best restaurants in the world, our media can’t.

  • Paul

    Merriam-Webster Dictionary:

    res·tau·rant noun ?res-t?-?ränt
    A business establishment where meals or refreshments may be purchased

    id·i·ot noun ?i-d?-?t
    A person who claims Federal Donut is not a restaurant

  • anon

    @ Paul above.

    Clever response but you are the idiot you so clearly define….and a quite disingenuous one.

    There are various classifications of restaurants as in casual (primarily) take out establishments and formal or informal sit down restaurants. The PHLMag list was about the 50 best places to go out and eat as in formal or informal sit down restaurants. Federal Donuts is a takeout joint with a few seats…..just like Pats, Genos, Johns, D’allesandros, Burger King, Wendys, Shake Shack and all manner of casual cheesesteak and fast food joint. That is not an opinion, it is a fact. Including Federal Donuts on the list therefore looks like a very obvious continuation of hype without any logic otherwise other casual “restaurants” would have been considered.

    Federal Donuts is a restaurant just like KFC and Krispy Kreme are restaurants.

  • Paul

    clue·less adjective ?klü-l?s
    Staring a fact in its face and refusing to recognize it

    waste (of time) verb ?w?st
    Unproductive effort trolling for hyperventilated reactions from people who get their panties all in a tangle over #50BestPHL; time that would be better spent going to Federal Donut

  • anon

    Yes federal donuts is a restaurant but the problem is the chicken is dry and has a texture of ham like it was overbrined.Way too salty too.If their signature item is inconsistent and they always run out I am not quite sure why it should be on the list.

  • m

    So when this list started formulating WILL BYOB wasn’t even open??

  • Brian Higgins

    I think people are going a little overboard here… Clearly this list is the reflection of one individual (or a small collection of individuals) who may (or may not) exhibit similar tastes in food, thereby incorporating some bit of bias into the Top 50 formulation. This should be so abundantly obvious that actually taking time to comment on it is an exercise in frivolity.

    Comments about statistically flawed are so off base it’s comical.

    If you all truly want a list that is reflective of the populace, pony up for a Zagat subscription.

  • John

    It’s a list put together by people who live in South Philly and work in center City. What do you expect?

  • B-C-D

    “until me and Art” ?????