Militants from the Shia Houthi group took control of the Jabal Ras area in the western Yemeni province of Al-Hudaydah on Monday.
"Dozens of armed Houthi gunmen have seized Jabal Ras and took over its security points, including the police headquarters, after the withdrawal of Al-Qaeda militants from the area," a security official, who asked to remain anonymous, told Anadolu Agency.
Government officials have yet to comment on the current situation in the area.
Al-Qaeda militants took over the area Saturday after fierce clashes with security forces left several soldiers dead.
They withdrew from the region Sunday night.
"Nearly 19 policemen were killed Saturday in an attack by Al-Qaeda militants that simultaneously targeted two checkpoints and the police headquarters," Yemen's Interior Ministry said on its website.
Security remains a major challenge in the southern Arabian nation, which has been beset by political instability since a popular uprising ended the long rule of ex-president Ali Abdullah Saleh in 2012.
In recent months, Yemen seemed to be teetering on the edge of civil war as the powerful Shia Houthi group sought to expand its control beyond capital Sanaa, which they captured in late September.
The growing Houthi influence in the country has pitted the Shia movement against Al-Qaeda, which remains active in Yemen.
Fighting between the two sides has left dozens dead in recent days in the country's central and western regions.
