Chief U.S. District Judge Vaughn Walker in San Francisco ruled on Wednesday that California's Proposition 8 violates the U.S. Constitution.
He however granted a temporary stay, which stops his decision from taking immediate effect.
Supporters of Proposition 8 argued, prior to Walker's ruling, that same-sex marriages would be performed soon after his decision and could be complicated by rulings and appeals farther down the legal road.
The judge's decision striking down the ban handed supporters of gay rights a major victory in a case that both sides say is sure to wind up before the U.S. Supreme Court.
The 136-page opinion is an initial step in what will likely be a lengthy fight over the proposition, which defines marriage as a union between a man and a woman.
At stake in the trial was whether California's ban on same-sex marriage violates gay couples' rights to equal protection and due process, as protected by the U.S. Constitution.
The high-profile case is being watched closely by both supporters and opponents of same-sex marriage, as many say it is destined to make its way to the U.S. Supreme Court.
If it does, the case could result in a landmark decision on whether people in the United States are allowed to marry people of the same sex.
Same-sex marriage is currently legal in five U.S. states and in the District of Columbia, while civil unions are permitted in New Jersey. The five states are Massachusetts, Connecticut, Vermont, Iowa and New Hampshire.
"Proposition 8 fails to advance any rational basis in singling out gay men and lesbians for denial of a marriage license. Indeed, the evidence shows Proposition 8 does nothing more than enshrine in the California Constitution the notion that opposite-sex couples are superior to same-sex couples," Walker, who was appointed to the federal bench by former President Ronald Reagan, wrote in his opinion.
"Race restrictions on marital partners were once common in most states but are now seen as archaic, shameful or even bizarre," he added. "Gender no longer forms an essential part of marriage; marriage under law is a union of equals."
Elated supporters of same-sex marriage gathered to celebrate the judge's opinion in the Castro district of San Francisco.
After speeches and songs, they began a march to city hall.
People waved rainbow flags and U.S. flags, and carried signs that read, "We all deserve the freedom to marry," and "Separate is Unequal."
Similar rallies unfolded in cities across California -- including Los Angeles and San Diego.
"For our entire lives, our government and the law have treated us as unequal. This decision to ensure that our constitutional rights are as protected as everyone else's makes us incredibly proud of our country," said Kristin Perry, a plaintiff.
Perry and Sandy Stier, along with Jeffrey Zarrillo and Paul Katami, are the two couples at the heart of the case, which if appealed would go next to the 9th U.S. Circuit Court of Appeals.
Opponents of same-sex marriage have said their best bet lies with higher courts, and vowed to appeal the federal judge's ruling.
In a national survey, conducted by Gallup in May, 53 percent of respondents said same-sex marriages should not be recognized by law, while 44 percent said they should.
Proposition 8 is part of a long line of seesaw rulings, court cases, debates and protests over the controversial issue of same-sex marriage. It passed in California with some 52 percent of the vote in November 2008.
"Big surprise! We expected nothing different from Judge Vaughn Walker, after the biased way he conducted this trial," said Brian Brown, president of the National Organization for Marriage. "With a stroke of his pen, Judge Walker has overruled the votes and values of 7 million Californians who voted for marriage as one man and one woman."
