To apply mensural signs to a score you need to add a conventional time signature first, then modify its display by substituting with the appropriate mensural symbol in the Time signature properties dialog. Note that these signs are for show only; it is not possible to modify the duration ratio between different note values, and the actual underlying time signature remains unchanged.
[To be added]
add from symbols palette, hide
Early music, such as renaissance choral music, may be written without barlines or a time signature. This can be achieved in MuseScore as follows:
Set up the instruments or voices of the score in the usual way (See Setting up your score);
Hide the time signature;
in Format→Style→Score, check the box titled "Display note values across measure boundaries";
Right-click on a measure in the top staff and select Staff/Part properties;
Uncheck "Show barlines"; repeat this for each applicable staff (see note);
Click OK.
Note: In step 4 you can make the barlines invisible instead by selecting all and pressing V.
Note that the score still behaves as if measures are present. Measure properties can be applied if required.
Mensurstrich is a form of modern notation of early music in which barlines are drawn between staves rather than across them:
Right click on the bottom staff of the intended mensurstrich system, and select Staff/Part properties. Uncheck "Show barlines".
Select the first (not the start!) barline for each of the remaining staves;
Check the "Span to next staff" box in the Barline section of the Properties panel;
Adjust "Span from" to set the position of the top of each barline;
Press Set as staff default;
From the menu bar, select Format→Style→Score;
Check "Display note values across measure boundaries".