I stayed in the house with my wife and son for two weeks. Overall we had a great vacation but without some work to the house, I would look elsewhere for next summer in Montauk.
While the location is good, the house still needs a lot of work for it be worth going back too. If you are a group of friends versus a family, it is probably more manageable especially if the duration of stay is only a long weekend and you really won't be spending any time at the house however we were here for two weeks.
Pros:
-House is in a great neighborhood in Montauk and is a five minute walk to the beach
-Private beach parking pass which is essential with little kids
-Good wrap around deck for outdoor dining (although you do hear Montauk highway)
-Lots of yard space for our son to walk around
-Owner is very responsive and had a cleaning crew come back when we asked
Cons:
-The bedrooms were in really rough shape and quite frankly the worst part of the house. The upstairs hasn't been renovated so the appearance was already in rough shape. Even without a renovation, it would be a lot better if there were good beds (one bed was broken/squeaking), working blinds in the rooms (we had to use portable shades), and usable dressers (one bedroom didn't have a dresser.) Also one of bedroom doors doesn't shut so someone doesn't get privacy.
-The outdoor shower is just a shower head with no walls so it is not a true outdoor shower.
-The house needed a little TLC, one of the kitchen cabinets was hanging, there were no screens on some windows so you couldn't open them at night etc..
-The living room and kitchen are actually in decent shape for lounging around but the furniture could really use some updating so it doesnt skeeve people out
All in all, if they put some work into the house, even just better furniture and beds, it would be significantly better. Without, the location isn't good enough to bypass the downside to come back again.