Your 2-layer LSTM language model on Penn Treebank has 1,500 hidden units, a 10,000-word vocabulary and 66M parameters, 30M of them in the input embedding and the output softmax matrix.
Training perplexity is 38 and validation is 82, and you want better validation perplexity without adding parameters.