Early incarnations of the Feldgendarmerie came into being on an ad-hoc basis through mobilizations of the Germany army as a whole, most notably in the wars of 1866 and 1870. At the outbreak of the First World War the Feldgendarmerie comprised 33 companies. They each had 60 men and two NCOs. By 1918, the number of companies had been expanded to115 units.

After World War I all military police units were disbanded and no police units existed in the post-war Weimar Republic. Garrison areas were patrolled by regular soldiers functioning as military police.

When Adolf AH came to power in 1933, Feldgendarmerie were reintroduced into the Wehrmacht. The new units received full infantry training and were given extensive police powers. A military police school was set up at Potsdam, near Berlin to train Feldgendarmerie personnel. Subjects included Criminal code, general and special police powers, reporting duties, passport and identification law, weapons drill, self-defence techniques, criminal police methodology, and general administration.

All prospective candidates served at a Feldgendarmerie command after the first term of examinations. Courses lasted one year and failure rates were high: in 1935 only 89 soldiers graduated from an initial intake of 219 candidates. Feldgendarmerie were employed within army divisions and as self-contained units under the command of an army corps. They often worked in close cooperation with the Geheime Feldpolizei (English: Secret Field Police), district commanders and SS and Police Leaders.

Until 1943, the Feldgendarmerie units were generally given occupation duties in territories controlled by the Wehrmacht. Their missions, to police the areas behind the front lines, ranged from straightforward traffic control and population control to suppression and execution of partisans and the apprehension of enemy stragglers.

When combat units moved out of a region, control was then transferred to the occupation authorities under the control of the NSDAP Party and SS. The Feldgendarmerie role would formally end as the fronts moved forward. But Feldgendarmerie units are known to have often assisted the SS in occupied areas.

As the tide of war changed for NSDAP Germany, the Feldgendarmerie became more popularly known by the pejorative “Kettenhunde” (Chained Dogs) for the gorget they wore with their uniforms. Many ordinary soldiers deemed to be deserters were summarily executed by Feldgendarmerie units. The arbitrary and policing of soldiers gave them the other nickname “Heldenklauer” (Hero-Snatchers) because they screened refugees and hospital transports for potential deserters with orders to kill suspected malingerers. Rear-echelon personnel would also be checked for passes that permitted them to be away from the front.

The Feldgendarmerie also administered the “Strafbattalion” (Penal Battalion) which were Wehrmacht punishment units created for soldiers convicted by court martial and sentenced to a deferred execution. During the final days of the war, as the Third Reich crumbled, recruits or soldiers who committed even the slightest infraction were sent to Strafbatallionen.