While setting up Forms based authentication I ran into this error:
The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.
Looking around I came across the following: http://msdn.microsoft.com/en-us/library/ff649234.aspx
Go to your server’s IIS and open Sites | SharePoint Web Services | SercurityTokenServiceApplication.
Open that site’s web.config and add above the serviceMetadata element in the Behavior list.
Now when you login, you should get a more useful error. In my case it was:
You must specify a non-autogenerated machine key to store passwords in the encrypted format. Either specify a different passwordFormat, or change the machineKey configuration to use a non-autogenerated decryption key.
So I added new Machine keys to Admin, 80 and Web Services sites. Then it worked.
Aboutdev's Weblog 
Hi, I tried your workaround replacing the existing machine key in Admin site and my web app site. Also I added a new machine key to the Web services site. I don’t get that exception anymore but I always get a wrong Log in (like wrong password) from the log in method that I use SPClaimsUtility.AuthenticateFormsUser(Context.Request.UrlReferrer, txtEmail.Text, txtPassword.Text) that returns false insted of the exception.
Can you specify if you added machine keys without deleting existing machine keys?
Can you specify what method did you use to authenticate users?
Hi i have FBA auth using Sql Membership Provider and i want to user PasswordFormat=Encrypated and i am getting above error can you please advise how can i solve this ? i can see machine key entry in web.config files of central admin and application webapplication but not in STS .do i need to make entry in STS
Try add it to STS. Not sure what your setup is so if it doesn’t work, you can always undo. You may be better off asking on Stackoverflow.com since more people pay attention to issues on there.