Place potatoes in a large saucepan and cover with 2 inches of water. Bring to a simmer over medium-high heat. Reduce heat to medium and cook, partially covered, until tender, 10 to 12 minutes. Drain and transfer to a large bowl.
Meanwhile, cook bacon in a skillet over medium heat until crisp. Drain on a paper towel. Crumble when cool enough to handle.
Mash the potatoes, milk (or buttermilk), salt and pepper with a potato masher until the liquid is incorporated but some chunks of potato still remain.
Fold in sour cream, green onions and crumbled bacon.